Image processing apparatus, image processing method, and program

ABSTRACT

There is provided an image processing apparatus including a display image generation section configured to generate display image data by performing a display projection process in a case where panorama image data to be a display target is judged to be a full circumference panorama image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.14/402,760, filed Nov. 21, 2014, which is a national phase entry under35 U.S.C. §371 of International Application No. PCT/JP2013/060183 filedApr. 3, 2013, published on Dec. 12, 2013 as WO 2013/183347 A1, whichclaims priority from Japanese Patent Application No. JP 2012-129101filed in the Japanese Patent Office on Jun. 6, 2012.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, animage processing method, and a program for implementing these, andspecifically relates to an image process appropriate for the display ofpanorama image data.

BACKGROUND ART

As shown in JP 2010-161520A an image process which generates onepanorama image from a plurality of images is well known.

For example, a user obtains a large number of captured images (frameimage data) while horizontally sweeping a camera, and by combiningthese, a so-called panorama image can be obtained.

Note that, a “sweep” is said to be the operation of imaging directionmovement due to a rotational motion of an imaging apparatus at the timeof imaging, in order to obtain a plurality of frame image data forpanorama image generation. For example, in the case of moving an imagingdirection in a horizontal direction, a sweep direction will point to thehorizontal direction.

SUMMARY OF INVENTION Technical Problem

In the case where reproduction and display is performed for a panoramaimage, it is common to perform a so-called single view display or toperform a scroll display.

A single view display is displaying by compressing all panorama imagedata as one still image. In this case, while an entire panorama imagecan be viewed, it will be difficult to become a powerful image due tocompression.

A scroll display is displaying while scrolling by sequentially cuttingout parts of panorama image data. By a scroll display, a panorama imagecan be viewed on a screen larger for each portion.

However, for example, a scroll display is displaying by simplysequentially cutting out panorama image data projected onto acylindrical surface or the like, and a scene from a user viewpoint willnot necessarily be accurately reflected, such as a straight line portionbeing significantly curved such as the horizon or a building, forexample.

On the other hand, there is a reproduction technique in which a highsense of immersion and sense of presence are obtained, by setting apanorama image projected onto a cylindrical surface or the like to adisplay image by re-projecting onto a plane surface screen. Such atechnique of reproduction and display will be called a “projectiondisplay” in the description.

For example, QuickTime VR. An Image-Based Approach to VirtualEnvironment Navigation (Apple Computer, Inc.) or the like is well known.

While it can be considered that the degree of satisfaction of a user whoviews the display of a panorama image can be improved when using such aprojection display, there will be a difficulty in the following point ofsimply performing a projection display instead of a scroll display.

While a viewing angle of a panorama image may be necessary in authoringfor performing a projection display, there is usually little for a userto accurately know the viewing angle of a panorama image projected in apanorama mode of a digital still camera. For example, there a varioustypes of images, such as a full circumference image with a 360° viewingangle, from an image with a viewing angle of approximately 120° or animage with a viewing angle of approximately 180°, in a panorama image.Also, apart from a 360° viewing angle, this calculation for obtaining aviewing angle from panorama image data will become inaccurate due toinfluences such as alignment errors, the sweep radius or distortionaberrations, and an accurate viewing angle will not be able to beobtained.

In this way, a viewing angle will not be known by only panorama imagedata, and authoring of a projection display will not be able to beperformed. For example, to ensure that a viewing angle for a projectiondisplay is known, it may be necessary for a user to roughly estimate aviewing angle at the time of panorama imaging, and to manually input theviewing angle at the time of authoring. This will become a difficultoperation for an ordinary user. Further, when an input viewing angle issignificantly different than an actual one, it will result indistortions occurring in a projection display image.

From this, a projection display will not necessarily always be anoptimum display system from the viewpoint of actual use or from theviewpoint of display quality.

The present disclosure aims to implement an image process in order to beable to manually and accurately execute a display of a panorama image,in particular, a high-quality projection display, for example, by widelyassuming from specialist use or business use up to a device for ageneral consumer.

Solution to Problem

An image processing apparatus according to the present disclosureincludes a display image generation section configured to generatedisplay image data by performing a display projection process in a casewhere panorama image data to be a display target is judged to be a fullcircumference panorama image.

An image processing method according to the present disclosure includesgenerating display image data by performing a display projection processin a case where panorama image data to be a display target is judged tobe a full circumference panorama image.

A program according to the present disclosure causes a calculationprocessing apparatus to execute the process of the image processingmethod.

In such a present disclosure, in the case where it is judged to be afull circumference panorama image, when performing reproduction anddisplay of panorama image data, a display process (projection display)can be performed which performs a display projection process.

There will be cases where a full circumference panorama image can bejudged to have a viewing angle of 360° (or approximately 360°). That is,the viewing angle can be determined, when performing a projectiondisplay. Therefore, a process for a projection display as a viewingangle of 360° can be performed without imposing a load on a user, anddisplay image data for a high-quality projection display can beobtained.

Note that, for example, in the case where panorama image data of adisplay target is not a full circumference panorama image, a selectioncan be considered in which a display is performed by another displaysystem, or a display is not performed.

Advantageous Effects of Invention

According to the present disclosure, in the case where panorama imagedata is a full circumference panorama image, display image data isgenerated by performing a display projection process. Therefore, aprocess for a projection display as a panorama image with a 360° viewingangle can be performed without imposing an operation load such asviewing angle input on a user. As a result, it becomes possible to havea high-quality projection display with no distortions.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram of panorama imaging performed in anembodiment of the present disclosure.

FIG. 2 is an explanatory diagram of panorama image combination performedin an embodiment.

FIG. 3 is an explanatory diagram of a panorama image of an embodiment.

FIG. 4 is an explanatory diagram of a combination projection process anda display projection process of an embodiment.

FIG. 5 is an explanatory diagram of a scroll display and a projectiondisplay of an embodiment.

FIG. 6 is an explanatory diagram of a projection display of a 360°panorama image of an embodiment.

FIG. 7 is an explanatory diagram of a single view display and a listdisplay of an embodiment.

FIG. 8 is a block diagram of an image processing apparatus of anembodiment.

FIG. 9 is a block diagram of configuration examples of an imageprocessing apparatus mounting device of an embodiment.

FIG. 10 is a block diagram of configuration examples of an imageprocessing apparatus mounting device of an embodiment.

FIG. 11 is an explanatory diagram of a first configuration example of anembodiment.

FIG. 12 is a flow chart of processes in the first configuration exampleof an embodiment.

FIG. 13 is an explanatory diagram of a second configuration example ofan embodiment.

FIG. 14 is a flow chart of processes in the second configuration exampleof an embodiment.

FIG. 15 is an explanatory diagram of a third configuration example of anembodiment.

FIG. 16 is a flow chart of processes in the third configuration exampleof an embodiment.

FIG. 17 is an explanatory diagram of a fourth configuration example ofan embodiment.

FIG. 18 is a flow chart of processes in the fourth configuration exampleof an embodiment.

FIG. 19 is a block diagram of an imaging apparatus of a fifthconfiguration example of an embodiment.

FIG. 20 is a block diagram of a functional configuration of a panoramacombination section and an association section installed in the imagingapparatus of an embodiment.

FIG. 21 is an explanatory diagram of panorama images, in the case wherea viewing angle is not 360°, and to which alignment is not performed.

FIG. 22 is an explanatory diagram of a panorama image to which alignmentis performed of an embodiment.

FIG. 23 is an explanatory diagram of alignment of a panorama image of anembodiment.

FIG. 24 is a flow chart of processes at the time of panorama imaging ofan embodiment.

FIG. 25 is an explanatory diagram of display mode transition of anembodiment.

FIG. 26 is a block diagram of a functional configuration of a displayimage generation section installed in the imaging apparatus of anembodiment.

FIG. 27 is a flow chart of a display process of an embodiment.

FIG. 28 is a block diagram of a computer apparatus of an embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, the embodiments will be described in the following order.

Note that, a device having the display image generation section 20 inthe embodiments will become the image processing apparatus in theclaims.

-   <1. Outline of panorama combination and reproduction display>-   <2. Image processing apparatus configuration examples>-   <3. First configuration example (metadata)>-   <4. Second configuration example (dedicated program)>-   <5. Third configuration example (file link information)>-   <6. Fourth configuration example (dedicated format)>-   <7. Fifth configuration example (application example to an imaging    apparatus)>-   (7-1: Configuration of the imaging apparatus)-   (7-2: Panorama combination and association processes)-   (7-3: Panorama image display)-   <8. Sixth configuration example (application example to a computer    apparatus and program)>-   <9. Modified example>

<1. Outline of Panorama Combination and Reproduction Display>

First, an outline of panorama combination will be described.

An imaging apparatus 50 of an embodiment, which will be described later,or a general imaging apparatus of recent years (a digital still camera),can generate a panorama image by performing a combination process for aplurality of still images (frame image data) obtained by a photographerwho is performing imaging while rotatably moving the imaging apparatus50 around some rotation axis.

In particular, a full circumference panorama image with a viewing angleof 360° can be generated by the imaging apparatus 50 of an embodiment.

FIG. 1A shows the movement of the imaging apparatus 50 at the time ofpanorama imaging. Since unnaturalness of junctions will occur whenparallaxes of the background and foreground are combined at the time ofpanorama imaging, it is desirable for a rotation centre at the time ofimaging to be set to a rotation center of a point of a lenscharacteristic, at which parallax called nodal point (Nodal Point) isdoes not occur. Rotational movement of the imaging apparatus 50 at thetime of panorama imaging will be called a “sweep”.

FIG. 1B is a schematic diagram at the time when an appropriate alignmentis performed for a plurality of still images obtained by a sweep of theimaging apparatus 50. In this figure, each still image obtained byimaging is shown in a time sequence of imaging. That is, frame imagedata captured from a time 0 up to a time (n−1) is set to frame imagedata FM #0, FM#1 . . . FM#(n−1).

In the case where a panorama image is generated from n still images, acombination process is performed for a series of n frame image data FM#0to FM#(n−1) continuously captured such as that illustrated.

As shown in this FIG. 1B, since it may be necessary for each capturedframe image data to have overlapping portions with adjacent frame imagedata, it is necessary for an imaging time interval of each frame imagedata of the imaging apparatus 50, and an upper-limit value of a speed atwhich a photographer performs a sweep, to be appropriately set.

Also, since a frame image data group aligned in this way will have manyoverlapping portions, it may be necessary to determine a region to beused for a final panorama image for each frame image data. In otherwords, this is the same as determining a junction (seam: seam) of imagesin a panorama combination process.

In FIG. 2A and FIG. 2B, examples of seams SM are shown.

In a seam, it can be set to a vertical straight line in a sweepdirection such as shown in FIG. 2A, or a non-straight line (curved lineor the like) such as shown in

In this FIG. 2A and FIG. 2B, a seam SM0 is shown as a junction betweenframe image data FM#0 and FM#1, a seam SM1 is shown as a junctionbetween frame image data FM#1 and FM#2, . . . and a seam SM(n−2) isshown as a junction between frame image data FM#(n−2) and FM#(n−1).

Note that, by having these seams SM0 to SM(n−2) set to junctions betweenadjacent images at the time of combining, the portions which have beenmade shaded parts in each frame image data will become image regionswhich are not used for a final panorama image.

Further, when performing panorama combination, an objective is set forreducing unnaturalness of images in the vicinity of the seams, and therewill be cases where blend processing is also performed for image regionsin the vicinity of the seams. Also, there will be cases where commonportions of each frame image data are connected by blend processing in awide range, or there will be cases where pixels contributing to thepanorama image are selected for each pixel from the common portions, andin these cases, while junctions will clearly be present, these widerange connection portions will have the same appearance as seams for thepresent disclosure.

Further, as shown in FIG. 1B, as a result of the alignment of each frameimage data, some movement will be recognized not only in the generalsweep direction, but also in a direction perpendicular to that of thesweep. There will be gaps occurring by this due to camera shake of aphotographer at the time of the sweep.

By determining the seams of each frame image data, performingconnections by connection or blend processing at these boundary regions,and finally trimming unnecessary portions in a direction perpendicularto that of a sweep by considering a camera shake amount, a panoramaimage with a wide viewing angle can be obtained, with a sweep directionset to a long side direction, such as shown in FIG. 3, vertical linesshow seams, and a state is schematically shown in which a panorama imageis generated by respectively connecting n frame image data FM#0 toFM#(n−1) at seams SM0 to SM(n−2).

A specific operation example of panorama imaging, which is performed bya user for obtaining such a panorama image by using the imagingapparatus 50, will be described.

First to begin with, the imaging apparatus 50 is set to a panoramaimaging mode, and a maximum image viewing angle is additionally set. Forexample, 120°, 180°, 360° or the like is capable of being set as themaximum image viewing angle, and in the case where 360° is selected,so-called full circumference imaging can be performed which stores anentire scene of the surroundings for all photographs. Note that, anexpression such as “360° panorama imaging mode” is used in the casewhere the maximum image viewing angle is set to 360° in the panoramaimaging mode, and an expression such as “180° panorama imaging mode” isused in the case where the maximum image viewing angle is set to 180°.

Next, panorama imaging is started by having a photographer press ashutter button of the imaging apparatus 50, and a sweep operation isperformed. Afterwards, in the case where end conditions of panoramaimaging are satisfied, the panorama imaging will be completed. Forexample, the end conditions are as follows.

-   -   The case of reaching a maximum image viewing angle set in        advance.    -   The case of the sweep operation being stopped by a user.    -   The case of a user pressing the shutter button again.    -   The case of a user removing a finger from the shutter button        (for a specification of continuing to press the shutter button        during panorama imaging).    -   The case of some error occurring.

After imaging is completed, a panorama combination process isautomatically started, a combination of a panorama image is completedafter a certain time has elapsed, and this panorama image data is heldin a recording device. Afterwards, a list is usually made the same asthat of the still images, and can be displayed.

Generated panorama image data such as this and a display mode of thiswill be described.

FIG. 4A is a schematic diagram of a projection process at the time ofpanorama combination. While the processing systems at the time ofpanorama combination are diverse, a process is often performed, forexample, which projects onto a cylindrical surface, a spherical surface,a cube or the like. This will be called a “combination projectionprocess” in the description, and is distinguished from a “displayprojection process” which is performed for a projection display, whichwill be described later.

In this FIG. 4A, a large amount of frame image data FM, which iscaptured while sweeping at the time of panorama imaging, is shown. Itcan be said that frame image data FM, that is, a captured image the sameas that of normal photographing, projects a three-dimensional scene ontoa plane surface (for example, an imager plane surface of a camera). Whenpanorama image data such as that of FIG. 3 is generated from such alarge amount of frame image data FM, a state will be set in which eachframe image data FM is projected onto a cylindrical surface FR, forexample, as a combination projection process. In this diagram, projectedframe image data FMpd is schematically shown along a cylindrical surfaceFR.

Panorama image data PD such as that of FIG. 4B is generated byconnecting the frame image data FMpd to which such a combinationprojection process has been performed. The panorama image data PD can beimage data to which projection has been performed again on thecylindrical surface FR as seen from a projection center CT.

In the present embodiment, a single view display, a list display, ascroll display and a projection display are capable of being executed asdisplay modes for such panorama image data PD. The display modes areillustrated in FIG. 5, FIG. 6 and FIG. 7.

FIG. 5 schematically shows the concepts of a scroll display and aprojection display.

A scroll display cuts out parts of the panorama image data PD, which isan image projected onto a cylindrical surface or the like such as thatdescribed above, as they are, and displays by scrolling up, down, leftand right automatically or in accordance with a manual operation.

On the other hand, a projection display is different to that of a scrolldisplay, and displays an image to which projection (a display projectionprocess) is performed again onto a virtual plane surface screen VS forpanorama image data PD projected onto a cylindrical surface, a sphericalsurface, cube or the like.

That is, in this case, a process is performed which projects, onto aplane surface, panorama image data projected onto a projection surfaceat the time of panorama image generation as a display projectionprocess. Also, an image projected onto a plane surface is set to displayimage data.

In this way, an image at the time of imaging can be reproduced,projection distortions caused by cylindrical projection or sphericalsurface projection as seen from a scroll display are eliminated, and adisplay with an improved sense of immersion and sense of presence isachieved.

For example, FIG. 6 schematically shows a projection display of panoramaimage data PD obtained in a 360° panorama imaging mode. Panorama imagedata PD is image data projected onto a 360° cylindrical surface FR ofthe surroundings of a projection center CT. This panorama image data PDis projected onto virtual plane surface screens VSO . . . VS45 . . . orthe like corresponding to each angle. For example, the plane surfacescreen VSO is shown as a plane surface screen centered on a 0° directionfrom the projection center CT, and the plane surface screen VS45 isshown as a plane surface screen centered on a 45° direction from theprojection center CT. In the figure, each of the plane surface screensVS135, VS180, VS225 and VS315 of the 135° direction, the 180° direction,the 225° direction and the 315° direction are additionally illustrated.

In a projection display, an image projected onto the plane surfacescreen VS set to such a viewing direction from the projection center CTis displayed. By appropriately setting a vector or distance from theprojection center CT to the plane surface screen VS automatically or inaccordance with a manual operation, a change of viewing direction or azoom-in/zoom-out are possible.

For example, if a projection image of the plane surface screen VSO, andprojection images of the plane surface screen VS1 (1° direction), theplane surface screen VS2 (2° direction) . . . the plane surface screenVS45 . . . the plane surface screen VS135 . . . the plane surface screenVS315 . . . and the plane surface screen VS0 are sequentially displayed,it becomes possible to display so as to look over the full circumferencethe surroundings from the projection center CT.

Further, the display image will be zoomed-in/zoomed-out, by moving theposition of the plane surface screen VS far from the projection centerCT or bringing it close to the projection center CT.

However, apart from a full circumference panorama image captured in a360° panorama imaging mode, a viewing angle of panorama image data willnot be able to be accurately obtained. Also, a three-dimensional modelat the time of performing a display projection process will not able tobe strictly defined. Accordingly, that other than 360° panorama imagedata will not be appropriate for a projection display.

Next, FIG. 7A shows an example of a single view display, and FIG. 7Bshows an example of a list display.

Since a single view display generates display image data which simplyincludes all the panorama image data PD, and displays this, it is anormal display operation. For example, by having a panorama image becomea horizontally long image, and combining it with a different aspectratio of a display screen, parts on the screen such as shown in FIG. 7A(in the case of this example, an upper region and a lower region) willoften become regions which are not displayed.

As shown in FIG. 7B, a list display displays image data capable of beingreproduced and displayed in a list. For example, by having a userperform image specification in a list display, panorama image data PD towhich the above described scroll display or projection process isperformed can be selected.

Note that, for example, while the images of the list display of FIG. 7Bare not images in which entire horizontally long panorama images arecompressed, in the case where panorama image data PD is represented in alist display, parts of a panorama image may be cut out and compressed,and a display may be performed such as that illustrated. It goes withoutsaying, for example, that there will also be list display modes such asa plurality side by side, by compressing the entire horizontally longpanorama images.

<2. Image Processing Apparatus Configuration Examples>

Configuration examples of an image processing apparatus of an embodimentwill be described.

FIG. 8A shows a configuration example having an image processingapparatus 1, which includes an association section 11 for generatedpanorama image data, and a panorama combination section 10. Theassociation section 11 and the panorama combination section 10 may beincluded in one body within a same device, or may be each included indifferent device bodies.

Further, FIG. 8B shows a configuration example having an imageprocessing apparatus 2, which includes a display image generationsection 20 for displaying panorama image data, and a display section 21.The display image generation section 20 and the display section 21 maybe included in one body within a same device, or may be each included indifferent device bodies.

In the configuration of FIG. 8A, the panorama combination section 10generates panorama image data PD from an input image group FMS. Theinput image group FMS is a series of frame image data FM obtained byhaving a photographer perform an imaging operation while sweeping in apanorama imaging mode. The panorama combination section 10 generatespanorama image data PD such as that described above in FIG. 1 to FIG. 3,by using a plurality of frame image data FM as the input image groupFMS.

Then, the panorama combination section 10 supplies the generatedpanorama image data PD to the association section 11. Note that, adetailed processing example of the panorama combination section 10 willbe stated in the description of a fifth configuration example of anembodiment.

Further, the panorama combination section 10 also supplies various typesof panorama processing information Ip to the association section 11 inthe generation process of the panorama image data PD.

Panorama processing information Ip is the generic name for the varioustypes of information generated in the processes of the panoramacombination process.

For example, information which shows a mode type such as a 360° panoramaimaging mode or a 180° panorama imaging mode (hereinafter, panorama modetype information) can be considered as the content of the panoramaprocessing information Ip.

Further, information which shows a judgment result of whether or not thepanorama image data PD has a 360° viewing angle (hereinafter, 360°viewing angle judgment information) can be considered as the content ofthe panorama processing information Ip. For example, it is informationof a result in which it is judged whether or not it is an exactly 360°viewing angle. Note that, it may be a result in which it is judged tohave a roughly approximate 360° viewing angle.

Further, information which shows whether or not the panorama image dataPD has a 360° viewing angle, and that a both end alignment is performed(hereinafter, both end alignment information) can be considered as thecontent of the panorama processing information Ip. While both endalignment will be stated in the description of a fifth configurationexample of an embodiment, it can be simply said that it is a processwhich performs an adjustment so that there are no image gaps in thevertical direction and the horizontal direction at the 0° and 360°positions of the panorama image, that is, at both ends of the panoramaimage.

Further, information of a projection surface of a combination projectionprocess (hereinafter, combination projection surface information) can beconsidered as the content of the panorama processing information Ip. Forexample, it is information which shows a cylindrical surface, aspherical surface, a polygonal surface, a plane surface or the like.

Further, information which shows a horizontal viewing angle and avertical viewing angle of the panorama image data PD (hereinafter,horizontal and vertical viewing angle information) can be considered asthe content of the panorama processing information.

In the case where the generated panorama image data PD is judged to be afull circumference panorama image, the association section 11 performs aprocess (association process) which associates this panorama image dataPD with information with shows that this panorama image data PD is afull circumference panorama image. Specific process examples, as theassociation process, will be described in a first configuration exampleto a fourth configuration example of an embodiment.

Also, the association section 11 outputs panorama image data forrecording or transmitting. In order for a distinction in thedescription, panorama image data for recording or transmitting, which isoutput by the association section 11, will be called, in particular, “apanorama image file PDr”.

This panorama image file PDr is panorama image data which is recorded toa recording medium or is transmitted to another device. In particular,in the case where the panorama image data PD has an approximately 360°viewing angle (or an exactly 360° viewing angle), there will be one or aplurality of data files of a state in which information which shows itis a full circumference panorama image is associated.

To continue, in the configuration of FIG. 8B, the display imagegeneration section 20 captures the panorama image file PDr.

For example, the panorama image file PDr recorded to a recording mediumby being output from the above described association section 11 issupplied to the display image generation section 20 by being read fromthe recording medium. Alternatively, the panorama image file PDrtransmitted and output by being output from the association section 11is supplied to the display image generation section 20 by beingreceived.

For example, the display image generation section 20 generates displayimage data PDdsp for the above described single view display, listdisplay, scroll display or projection display. In particular, in thecase where a panorama image of the panorama image file PDr selected as adisplay target is judged to be a full circumference panorama image,display image data PDdsp for a projection display is generated byperforming a display projection process.

Also, the display image generation section 20 supplies the display imagedata PDdsp to the display section 21. Specific process examples of thedisplay image generation section 20 will be described in the firstconfiguration example to the fourth configuration example of anembodiment.

The display section 21 has display panel such as a liquid crystal panel,an organic Electroluminescence (EL) panel, a plasma display panel or aCathode Ray Tube (CRT), and a display drive section for this displaypanel, and performs various types of display. By having the displayimage data PDdsp supplied from the display image generation section 20a, a display operation based on this display image data PDdsp isexecuted in this display section 21. For example, a single view display,a list display, a scroll display or a projection display related to apanorama image is performed.

In the case of the present example, in the case where a panorama imageof the panorama image file PDr selected as a display target is judged tobe a full circumference panorama image, display image data PDdsp isgenerated by performing a display projection process by the displayimage generation section 20, and is supplied to the display section 21.For example, images projected onto the virtual plane surface screen VSof FIG. 6 are sequentially supplied, and in this case, a projectiondisplay is executed by the display section 21.

Various types of configuration examples can be considered as anelectronic device which has the image processing apparatus 1, whichincludes the association section 11, and the image processing apparatus2, which includes the display image generation section 20, such asdescribed above. Various types of apparatus configuration examples willbe illustrated in FIG. 9 and FIG. 10 as the electronic device.

FIG. 9A is an example in which the image processing apparatus 1 and theimage processing apparatus 2 are constituted in an apparatus DVC1 whichincludes an imaging function.

The apparatus DVC1 of this example has an imaging section 12, a panoramacombination section 10, an association section 11, a recording andreproduction section 13, a display image generation section 20, and adisplay section 21.

For example, the imaging section 12 obtains a series of a plurality offrame image data FM for generating a panorama image, by an imagingoperation of panorama image data. This plurality of frame image data FMis supplied to the panorama combination section 10 as an input imagegroup FMS. The panorama combination section 10 generates panorama imagedata PD from the input image group FMS, and outputs panorama image dataPD and panorama processing information Ip. The association section 11performs the above described association process, and outputs a panoramaimage file PDr. The panorama image file PDr is recorded to a recordingmedium in the recording and reproduction section 13.

The panorama image file PDr read from the recording medium in therecording and reproduction section 13 is supplied to the display imagegeneration section 20, and display image data PDdsp is generated. Thisdisplay image data PDdsp is supplied to the display section 21, and adisplay such as a projection display is executed.

In this way, for example, the apparatus DVC1 can be assumed to be adigital still camera, a video camera, a mobile phone unit, a computerapparatus or the like, which includes a panorama imaging function, arecording and reproduction function, and a display function.

FIG. 9B is an example in which the image processing apparatus 1 and theimage processing apparatus 2 are constituted in an apparatus DVC2 whichincludes an image data input function from the outside.

The apparatus DVC2 of this example has an input section 14, a panoramacombination section 10, an association section 11, a recording andreproduction section 13, a display image generation section 20, and adisplay section 21. That is, when compared to the above described FIG.9A, there will be cases where the input section 14 is included insteadof the imaging section 12.

For example, the input section 14 inputs a series of a plurality offrame image data FM for generating a panorama image, which is obtainedby an imaging operation of panorama image data in an external imagingapparatus. This plurality of frame image data FM is supplied to thepanorama combination section 10 as an input image group FMS. Theoperations of each section hereinafter are the same as those of FIG. 9A.

In this way, the apparatus DVC2 is a device which performs panoramaimage generation by using frame image data FM obtained in an externalimaging apparatus, or performs a display of a panorama image. Forexample, it can be assumed to be an image editing device, a recordingand reproduction apparatus, a television reception device, a mobilephone unit, a computer apparatus or the like used by wired or wirelesslyconnecting to an imaging apparatus such as a digital still camera.

An apparatus DV3 of FIG. 9C is an example in which the image processingapparatus 1 and the image processing apparatus 2 are constituted as adevice which does not have the display section 21.

The Apparatus DVC3 of this example has an input section 14 (or animaging section 12), a panorama combination section 10, an associationsection 11, a recording and reproduction section 13, and a display imagegeneration section 20. The display section 21 of this case becomes anexternal device connected to the apparatus DVC3. While the operations ofeach section are the same as those of FIG. 9A and FIG. 9B, the displayimage generation section 20 transmits the generated display image dataPDdsp to the external display section 21, and causes a display such as aprojection display to be executed.

In this way, the image processing apparatus 1 and the image processingapparatus 2 are capable of being executed as a device which does nothave a display function.

FIG. 10A is an example of an apparatus DVC4 which becomes the imageprocessing apparatus 1, and an apparatus DVC5 which becomes the imageprocessing apparatus 2.

The apparatus DVC4 has an imaging section 12 (or an input section 14), apanorama combination section 10, an association section 11, and arecording section 13R.

The recording section 13R records a panorama image file PDr from theassociation section 11 to a portable recording medium 90. It can beassumed to be a memory card, an optical disc, a magnetic disc or thelike as the portable recording medium 90.

The apparatus DVC5 has a reproduction section 13P, a display imagegeneration section 20, and a display section 21.

The reproduction section 13P reproduces a panorama image file PDr fromthe portable recording medium 90, and supplies it to the display imagegeneration section 20. The display image generation section 20 generatesdisplay image data PDdsp based on the panorama image file PDr, andcauses a display of a panorama image to be executed by supplying to thedisplay section 21.

In such an example, a configuration in which a panorama image file PDris captured by the portable recording medium 90 can be considered as aconfiguration in which the apparatus DVC4 which becomes the imageprocessing apparatus 1 and the apparatus DVC5 which becomes the imageprocessing apparatus 2 are different bodies. Note that, an example inwhich the external display section 21 is used can also be considered,which does not include the display section 21 in the apparatus DVC5.

FIG. 10B is an example of an apparatus DVC6 which becomes the imageprocessing apparatus 1, and an apparatus DVC7 which becomes the imageprocessing apparatus 2.

The apparatus DVC6 has an imaging section 12 (or an input section 14), apanorama combination section 10, an association section 11, and acommunication section 15.

The communication section 15 transmits a panorama image file PDr fromthe association section 11 to the outside.

The apparatus DVC7 has a communication section 16, a display imagegeneration section 20, and a display section 21.

The communication section 16 receives the panorama image file PDrtransmitted from the apparatus DVC6, and supplies it to the displayimage generation section 20. The display image generation section 20generates display image data PDdsp based on the panorama image file PDr,and causes a display of a panorama image to be executed by supplying tothe display section 21.

In such an example, a configuration in which a panorama image file PDris delivered by communication can also be considered as a configurationin which the DVC6 which becomes the image processing apparatus 1 and theDVC7 which becomes the image processing apparatus 2 are differentbodies.

Communication between the communication sections 15 and 16 can also beconsidered for either wired communication or wireless communication.Further, it will be assumed that the communication is a so-calledcommunication mode, such as communication by a cable connection betweendevices, communication by a Universal Serial Bus (USB) or the like,communication by a Local Area Network (LAN), or communication using apublic network such as the internet. In addition, the communicationsection 15 may perform communication which uploads a panorama image filePDr to a server or the like on the internet, or the communicationsection 16 may perform communication which downloads a panorama imagefile PDr from a server. Further, the communication sections 15 and 16may perform communication in which cloud computing is assumed.

Note that, an example in which the external display section 21 is usedcan also be considered, which does not include the display section 21 inthe apparatus DVC7.

Heretofore, while device configuration examples have been shown, thedevice configurations which implement the image processing apparatuses 1and 2 can be additionally considered for various types.

For example, in the case where a cloud computing system is assumed, theassociation section 11 and the display image generation section 20 canalso be considered to be arranged on a network and not in user usedevices.

Further, the above described panorama image file PDr is data to which aprocess is performed which associates panorama image data PD withinformation which shows that this panorama image data PD is a fullcircumference panorama image. While it can be considered that a file ismade in which information which shows that it is a full circumferencepanorama image (for example, metadata or a dedicated program, which willbe described later) is incorporated with the panorama image data PD asthis panorama image file PDr, additional panorama image data PD andinformation which shows that it is a full circumference panorama imagecan be included as different files, and there will also be a mode inwhich these are associated by some information.

<3. First Configuration Example (Metadata)>

Hereinafter, specific configuration examples as the association section11 and the display image generation section 20 shown in FIG. 8 will bedescribed. A first configuration example is an example in which theassociation section 11 performs an association process using metadata.

FIG. 11A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this firstconfiguration example, the association section 11 has a metadatageneration section 11 a and a data integration section 11 b.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The panorama processing information Ip is supplied to the metadatageneration section 11 a. The metadata generation section 11 a generatesmetadata MT reflecting the panorama processing information Ip. There ispanorama mode type information, 360° viewing angle judgment information,both end alignment information, combination projection surfaceinformation, horizontal and vertical viewing angle information or thelike as the content of the above described panorama processinginformation Ip.

In the case where the generated panorama image data PD is judged to be afull circumference panorama image, the panorama mode type information,the 360° viewing angle judgment information or the both end alignmentinformation from among these can become information which shows thatthis panorama image data PD is a full circumference panorama image.

The data integration section 11 b generates a panorama image file PDr byintegrating the panorama image data PD generated by the panoramacombination section 10 with the metadata MT generated by the metadatageneration section 11 a. For example, the panorama image file PDrbecomes a structure such as that of FIG. 11B. That is, it is a data filestructure which includes a header, panorama image data PD, and metadataMT.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr is recorded to arecording medium or is transmitted and output.

FIG. 11C shows the display image generation section 20 in the imageprocessing apparatus 2. In this first configuration example, the displayimage generation section 20 has a data separation section 20 a, ametadata interpretation section 20 b, and an image generation processingsection 20 c.

The panorama image file PDr, which is reproduced from the recordingmedium or is received, is input to the data separation section 20 a. Thedata separation section 20 a separates the metadata MT and the panoramaimage data PD from this panorama image file PDr.

The metadata MT is interpreted by the metadata interpretation section 20b, and is supplied to the image generation processing section 20 c aspanorama processing information Ip (panorama mode type information, 360°viewing angle judgment information, both end alignment information,combination projection information, horizontal and vertical viewingangle information).

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is determined whether or not to perform display image dataPDdsp generation for a projection display, in accordance with thecontent of the panorama processing information Ip. That is, in the casewhere the panorama image data PD of a display target is judged to be afull circumference panorama image, a process for a projection display isperformed, and in the case where it is not a full circumference panoramaimage, a process for a scroll display or a single view display, forexample, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21 and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 12.

FIG. 12A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F1, the panorama combination section 10 captures an input imagegroup FMS.

In step F2, panorama image data PD is generated by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the metadata generation section 11 a).

In step F3, the association section 11 (the metadata generation section11 a) generates metadata MT based on the panorama processing informationIp.

In step F4, the association section 11 (the data integration section 11b) integrates the panorama image data PD with the metadata MT, andgenerates, for example, a panorama image file PDr such as that of FIG.11B.

In step F5, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

In FIG. 12B, the processes of the display image generation section 20 ofthe image processing apparatus 2 are shown.

In step F51, the display image generation section 20 (the dataseparation section 20 a) captures the panorama image file PDr, which isreproduced by the recording medium which is not illustrated, or isreceived by the communication section which is not illustrated.

In step F52, the display image generation section 20 (the dataseparation section 20 a, the metadata interpretation section 20 b)separates the panorama image data PD and the metadata MT from thepanorama image file PDr, extracts the metadata MT, and obtains panoramaprocessing information Ip by interpreting this.

In step F53, the display image generation section 20 (the imagegeneration processing section 20 c) branches the processes by whether ornot the panorama image data PD of a display target of this time is afull circumference panorama image with a 360° viewing angle.

The judgment of whether or not the panorama image data PD is a fullcircumference panorama image can be considered for various types.

By referring to 360° viewing angle judgment information in the panoramaprocessing information Ip, it can be judged whether or not the panoramaimage data PD has an exactly 360° viewing angle. Accordingly, it may bejudged to be a full circumference panorama image in the case where ithas an exactly 360° viewing angle.

Further, by referring to both end alignment information in the panoramaprocessing information Ip, it can be judged whether or not the panoramaimage data PD has a 360° viewing angle, and a both end alignment processis performed. Accordingly, it may be judged to be a full circumferencepanorama image by setting, as a condition, having a 360° viewing angleand both end alignment being performed.

Further, by referring to panorama mode type information in the panoramaprocessing information Ip, it can be judged whether or not panoramaimage data PD is captured in a is captured in a 360° panorama imagingmode, that is, whether or not it has an approximately 360° viewingangle. Accordingly, it may be judged to be a full circumference panoramaimage if it is captured in a 360° panorama imaging mode.

Note that, it may not necessarily be said that the panorama image dataPD captured in a 360° panorama imaging mode has an exactly 360° viewingangle. This is because, in a 360° panorama imaging mode, while panoramaimage data PD is generated from frame image data FM obtained by a 360°sweep at the time of imaging, the sweep is performed by a user, and willnot necessarily become an exactly 360° viewing angle. There will becases where the user stops the sweep before reaching 360°. Therefore,the panorama image data PD captured in a 360° panorama imaging modebecomes an image estimated to have an approximately 360° viewing angle.

In step F53, any of the above described judgment techniques are used,and in the case where it is judged to be a full circumference panoramaimage, the display image generation section 20 (the image generationprocessing section 20 c) proceeds to step F54, generates display imagedata PDdsp by performing a display projection process, and in step F55,performs a process which outputs this display image data PDdsp. Theprocesses of these steps F54 and F55 are continued up to it becomes adisplay end, in step F56, by a user operation or an automatic process.In this way, in the display section, for example, a display so as tolook over the full circumference or a display of an arbitrary directionis executed as a projection display.

Here, in a display projection process for performing a projectiondisplay, information of a viewing angle of a panorama image may benecessary. In the case of the present example, the panorama image dataPD which becomes the process target of step F54 is judged to be a fullcircumference panorama image such as described above, and accordingly,the image generation processing section 20 c may perform a displayprojection process by setting the panorama image data PD to a viewingangle of 360°. In this way, a high-quality projection display in whichthere are few image distortions can be performed.

Further, in a display projection process, panorama image data PD, towhich projection is performed onto a projection surface (cylindricalsurface, spherical surface, polygonal surface or the like) at the timeof panorama combination, is re-projected onto a virtual plane surfacescreen VS. Accordingly, information of the projection surface of thepanorama image data PD may become necessary. In the present example, byreferring to combination projection surface information in the panoramaprocessing information Ip, the image generation processing section 20 ccan accurately perform a re-projection process to the plane surfacescreen VS.

Further, in a projection display, by changing a horizontal and verticalviewing angle of the virtual plane surface screen VS, a display can beperformed which looks over the surroundings in the horizontal andvertical directions the same as that of a scroll display. For example,by having a 360° viewing angle in the horizontal direction, whileunrestricted scrolling can be performed in the horizontal direction, itwill be restricted in the vertical direction. In this case, by referringto horizontal and vertical viewing angle information in the panoramaprocessing information Ip, the image generation processing section 20 ccan restrict a scroll capable range of the vertical direction.

Note that, horizontal and vertical viewing angle information may beinformation which is simply used for a restriction of a display range,without being used in the projection process itself to the plane surfacescreen VS for display. Therefore, accurate and high values of a viewingangle may be shown. At the time of the generation of panorama image dataPD, it may not be necessary to perform difficult calculations forobtaining highly accurate horizontal and vertical viewing angles.

In step F53, in the case where it is judged not be a full circumferencepanorama image, in step F57, the display image generation section 20(image generation processing section 20 c) performs a display image datageneration process of another system. For example, display image dataPDdsp is generated as a scroll display or a single view display. Then,in step F58, a process is performed which outputs display image dataPDdsp. The processes of these steps F57 and F58 are continued up to itis judged to be a display end, in step F59. In this way, in the displaysection, a scroll display or a single view display other than aprojection display is executed.

In the case where it is not a full circumference panorama image, aprojection display will not be able to be executed with a high quality,without the viewing angle being exactly known. Accordingly, a scrolldisplay or a single view display is performed.

In the first configuration example such as described above, in the imageprocessing apparatus 1, the association section 11 generates a panoramaimage file PDr by associating metadata MT including information whichshows that it is a full circumference panorama image with the panoramaimage data PD.

Combination projection surface information which shows the type ofprojection surface of the frame image data at the time of panorama imagegeneration, 360° viewing angle judgment information of the panoramaimage data PD, both end alignment information, panorama mode informationand horizontal and vertical viewing angle information are included inthe metadata MT.

On the other hand, in the image processing apparatus 2, the displayimage generation section 20 generates display image data by performing adisplay projection process at the time when the panorama image data PDis judged to be a full circumference panorama image, and causes aprojection display to be executed. The judgment of a full circumferencepanorama image uses information which is associated as the metadata MD.

By such a configuration, a projection display can be automaticallyexecuted for a full circumference panorama image in which an accurateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

Further, panorama mode type information, 360° viewing angle judgmentinformation, both end alignment information or the like are included asthe metadata MT, and by having these associated with the panorama imagedata PD, a judgment of whether or not it is a full circumferencepanorama image can be easily and accurately executed. That is, theselection of a display system becomes easy and accurate.

Further, by having combination projection surface information includedin the metadata MT, a display projection process can be accuratelyperformed.

<4. Second Configuration Example (Dedicated Program)>

A second configuration example of an embodiment will be described. Inthe second configuration example, the association section 11 of theimage processing apparatus 1 associates a display processing program (adedicated program) used for the display of a full circumference panoramaimage with the panorama image data PD, as information which shows thatit is a full circumference panorama image.

Further, the display image generation section 20 of the image processingapparatus 2 judges that it is a full circumference panorama image, byhaving the display processing program associated with the panorama imagedata PD, and generates display image data by performing a displayprojection process based on this display processing program.

FIG. 13A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this secondconfiguration example, the association section 11 has a displayprocessing program generation section 11 d and a data integrationsection 11 e.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The panorama processing information Ip is supplied to the displayprocessing program generation section 11 d. The display processingprogram generation section 11 d causes a display processing program PGto be generated based on the panorama processing information Ip. Byjudging whether or not the panorama image data PD is a fullcircumference panorama image, by any of panorama mode type information,360° viewing angle judgment information and both end alignmentinformation in the panorama processing information Ip, the displayprocessing program generation section 11 d causes a display processingprogram PG to be generated, in the case of a full circumference panoramaimage.

For example, since it is described in Flash (Adobe), QuickTimeVR (Apple)or HTML5, as the display processing program PG, reproduction and displayis possible by a wide range of display devices.

The data integration section 11 e generates a panorama image file PDr byintegrating the panorama image data PD generated by the panoramacombination section 10 with the display processing program PG generatedby the display processing program generation section 11 d. For example,the panorama image file PDr becomes a structure such as that of FIG.13B. That is, it is a data file structure which includes a header,panorama image data PD and display processing program PG.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr is recorded to arecording medium, or is transmitted and output. Note that, asillustrated as the panorama image file PDr, while the panorama imagedata PD may be set to a normal panorama holding form, and may add adisplay processing program PG, it may be set to a form which embeds thepanorama image data PD in a display processing program itself.

FIG. 13C shows the display image generation section 20 in the imageprocessing apparatus 2. In this second configuration example, thedisplay image generation section 20 has a data separation section 20 f,and an image generation processing section 20 c.

A panorama image file PDr, which is reproduced from a recording mediumor is received, is input to the data separation section 20 f. The dataseparation section 20 a separates the panorama image data PD and thedisplay processing program PG from this panorama image file PDr, andsupplies them to the image generation processing section 20 c.

Note that, in the case where the panorama image data PD is not a fullcircumference panorama image, the display processing program PG is notincluded in the panorama image file PDr, and therefore the dataseparation section 20 f supplies only the panorama image data PD to theimage generation processing section 20 c.

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, if the display processing program PG is present, display imagedata PDdsp generation is performed for a projection display, byperforming a process based on the display processing program PG. In thecase where the panorama image data PD is not a full circumferencepanorama image, and the display processing program PG is not extracted,a process for another display, for example, a scroll display or a singleview display, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC 7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21, and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 14.

FIG. 14A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F11, the panorama combination section 10 captures an input imagegroup FMS.

In step F12, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the display processing program generationsection 11 d).

In step F13, the association section 11 (the display processing programgeneration section 11 d) confirms the panorama processing informationIp, and performs a judgment of whether or not the generation panoramaimage data PD is a full circumference panorama image.

Similar to that stated in the first configuration example, the judgmentof whether or not it is a full circumference panorama image can beconsidered by a system which refers to 360° viewing angle judgmentinformation in the panorama processing information Ip, a system whichrefers to both end alignment information, a system which refers to apanorama mode type information or the like.

In the case where it is a full circumference panorama image, it proceedsfrom step F14 to F15, and the association section 11 (the displayprocessing program generation section 11 d) causes a display processingprogram PG to be generated.

In step F16, in the case where a display processing program PG isgenerated, the association section 11 (the data integration section 11b) integrates the panorama image data PD and the display processingprogram PG, and generates a panorama image file PDr such as that of FIG.13B, for example. In the case where a display processing program PG isnot generated, that is, in the case where it is not a full circumferencepanorama image, a panorama image file PDr which includes panorama imagedata PD is generated.

In step F17, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

In FIG. 14B, the processes of the display image generation section 20 ofthe image processing apparatus 2 are shown.

In step F60, the display image generation section 20 (the dataseparation section 20 f) captures the panorama image file PDr, which isreproduced in the recording reproduction section which is notillustrated, or is received in the communication section which is notillustrated.

In step F61, the display image generation section 20 (the dataseparation section 20 f) performs a separation process of the panoramaimage file PDr, and confirms whether or not a display processing programPG is included.

In step F62, the display image generation section 20 braches theprocesses in accordance with the presence or not of a display processingprogram PG

In the case where a display processing program PG is present, in stepF63, the display processing program PG is captured, and is set to astate in which it is used as a program for display image generation bythe image generation 20 c.

Then, the display image generation section 20 (the image generationprocessing section 20 c) proceeds to step F64, display image data PDdspis generated by performing a display projection process in accordancewith the display processing program PG, and in step F65, a process isperformed which outputs this display image data PDdsp. The processes ofthese steps F64 and F65 are continued up to it becomes a display end, instep F66, by a user operation or an automatic process. In this way, inthe display section, for example, a display so as to look over the fullcircumference, a display of an arbitrary direction or the like isexecuted as a projection display.

Note that, information of a projection surface of the panorama imagedata PD, in which the panorama image data PD has a 360° viewing angle,may be described in the display processing program PG, such ashorizontal and vertical viewing angle information for additional scrollrestriction.

In the case it is judged, in step F62, that a display processing programPG is not present, the display image generation section 20 (the imagegeneration processing section 20 c) performs, in step F67, a displayimage data generation process based on an existing display processingprogram. For example, when display processing programs are stored as ascroll display or a single view display in the image generationprocessing section 20 c, display image data PDdsp is generated as ascroll display or a single view display using these. Then, in step F68,a process is performed which outputs the display image data PDdsp. Theprocesses of steps F67 and F68 are continued up to it is judged to be adisplay end, in step F69. In this way, in the display section, a scrolldisplay or a single view display other than a projection display isexecuted. That is, a scroll display or a single view display isperformed in the case where it is not a full circumference panoramaimage.

In the second configuration example such as described above, in theimage processing apparatus 1, the association section 11 generates apanorama image file PDr by associating the display processing program PGfor executing a projection display corresponding to a full circumferencepanorama image with the panorama image data PD.

On the other hand, in the image processing apparatus 2, if a displayprocessing program PG is present, the display image generation section20 generates display image data PDdsp for a projection display byperforming a display projection process based on this display processingprogram PG.

By such a configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

<5. Third Configuration Example (File Link Information)>

A third configuration example of an embodiment will be described. In thethird configuration example, the association section 11 of the imageprocessing apparatus 1 performs a process which associates a data fileincluding information which shows that it is a full circumferencepanorama image with a data file including panorama image data PD.

Further, the display image generation section 20 of the image processingapparatus 2 confirms the data file associated from the data fileincluding the panorama image data PD, and judges whether or not it is afull circumference panorama image by acquiring information. Then,display image data is generated by performing a display projectionprocess in the case where it is a full circumference panorama image.

FIG. 15A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this third embodimentexample, the association section 11 has a metadata generation section 11a and an association information addition section 11 c.

The panorama combination section 10 generates panorama image data PD,and outputs panorama processing information Ip. The panorama processinginformation Ip (panorama mode type information, 360° viewing anglejudgment information, both end alignment information, combinationprojection information, horizontal and vertical viewing angleinformation or the like) is supplied to the metadata generation section11 a.

The metadata generation section 11 a generates metadata MT reflectingthe panorama processing information Ip.

As shown in FIG. 15B, the association information addition section 11 cgenerates a panorama image file PDr1 which includes the panorama imagedata PD, and generates a data file PDr2 which includes the metadata MT.In this case, file link information LK1 and LK2 are added to thepanorama image file PDr1 and the data file PDr2, respectively, and aremutually associated. The file link information LK1 is information whichspecifies the data file PDr2, and the file link information LK2 isinformation which specifies the panorama image file PDr1. Note that, thefile link information which specifies the other may be added to only oneof the panorama image file PDr1 and the data file PDr2.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr1 and data file PDr2are recorded to recording mediums, or are transmitted and output. Therecording mediums to which each of the files are recorded may bedifferent recording mediums. Further, each of the files may betransmitted to a different device, a network server or the like.

FIG. 15C shows the display image generation section 20 in the imageprocessing apparatus 2. In this third configuration example, the displayimage generation section 20 has an association information extractionsection 20 d, a response data acquisition section 20 e, a metadatainterpretation section 20 b, and an image generation processing section20 c.

The panorama image file PDr1, which is reproduced from the recordingmedium or is received, is input to the association informationextraction section 20 d. The association information extraction section20 d extracts the file link information LK1 from this panorama imagefile PDr1 and supplies it to the response data acquisition section 20 e,and supplies panorama image data PD to the image generation processingsection 20 c.

The response data acquisition section 20 e performs an access ACLK byusing the file link information LK1, and acquires the associated datafile PDr2. For example, the data file PDr2 is read by accessing aprescribed recording medium. Alternatively, the data file PDr2 isreceived by accessing a prescribed external device, network server orthe like. Then, if the response data acquisition section 20 e acquiresthe data file PDr2, metadata MT is extracted, and is supplied to themetadata interpretation section 20 b. That is, the metadata MTassociated with the panorama image data PD of a display target of thistime is obtained.

The metadata MT is interpreted by the metadata interpretation section 20b, and is supplied to the image generation processing section 20 c aspanorama processing information IP (panorama mode type information, 360°viewing angle judgment information, both end alignment information,composition projection surface information, horizontal and verticalviewing angle information).

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is judged whether or not to perform display image data PDdspgeneration for a projection display, in accordance with the content ofthe panorama processing information Ip. That is, in the case where thepanorama image data PD of a display target is judged to be a fullcircumference panorama image, a process for a projection display isperformed, and in the case where it is not a full circumference panoramaimage, a process for another display, for example, a scroll display or asingle view display, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21, and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 16.

FIG. 16A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F21, the panorama combination section 10 captures an input imagegroup FMS.

In step F22, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the metadata generation section 11 a).

In step F23, the association section 11 (the metadata generation section11 a) generates metadata MT based on the panorama processing informationIp.

In step F24, the association section 11 (the association informationaddition section 11 c) generates a panorama image file PDr1 whichincludes the panorama image data PD and the file link information LK1,or generates a data file PDr2 which includes the metadata MT and thefile link information LK2.

In step F25, the panorama image file PDr1 and the data file PDr2 arerecorded to a recording medium by a recording reproduction section whichis not illustrated, or are transmitted and output to an external device,a network or the like by a communication section which is notillustrated.

FIG. 16B shows the processes of the display image generation section 20of the image processing apparatus 2.

In step F71, the display image generation section 20 (the associationinformation extraction section 20 d) captures the panorama image filePDr1 reproduced by the recording medium which is not illustrated, orreceived by the communication section which is not illustrated.

In step F72, the display image generation section 20 (the associationinformation extraction section 20 d, the response data acquisitionsection 20 e, the metadata interpretation section 20 b) performs accessby using the link information LK1 extracted from the panorama image filePDr1, and acquires the data file PDr2. Then, the metadata MT isextracted from the data file PDr2, and panorama processing informationIp is obtained by interpreting this.

In step F73, the display image generation section 20 (the imagegeneration processing section 20 c) branches the processes by whether ornot the panorama image data PD of a display target of this time is afull circumference panorama image with a 360° viewing angle.

As described in the first configuration example, the judgment of whetheror not the panorama image data PD is a full circumference panorama imageis possible by any of panorama mode type information, 360° viewing anglejudgment information and both end alignment information in the panoramaprocessing information Ip.

In step F73, in the case where it is judged to be a full circumferencepanorama image, the display image generation section 20 (the imagegeneration processing section 20 c) proceeds to step F74, display imagedata PDdsp is generated by performing a display projection process, andin step F75, a process is performed which outputs this display imagedata PDdsp. The processes of these steps F74 and F75 are continued up toit becomes a display end, in step F76, by a user operation or anautomatic process. In this way, in the display section, for example, adisplay so as to look over the full circumference, a display of anarbitrary direction or the like is executed as a projection display.

Note that, in this case, since the panorama image data PD is a fullcircumference panorama image, the viewing angle used in the displayprojection process may be set to 360°.

Further, by referring to combination projection surface information inthe panorama processing information Ip, the image generation processingsection 20 c can accurately perform a re-projection process to a planesurface screen VS.

In the case where it is judged, in step F73, not to be a fullcircumference panorama image, the display image generation section 20(the image generation processing section 20 c) performs, in step F77, adisplay image data generation process of another system. For example,display image data PDdsp is generated as a scroll display or a singleview display. Then, in step F78, a process is performed which outputsthe display image data PDdsp. The processes of these steps F77 and F78are continued up to it is judged to be a display end, in step F79. Inthis way, in the display section, a scroll display or a single viewdisplay other than a projection display is executed. That is, in thecase where the panorama image data PD is not a full circumferencepanorama image, and the viewing angle is not accurately known, a scrolldisplay or a single view display is performed.

In the third configuration example such as described above, in the imageprocessing apparatus 1, the association section 11 generates a data filePDr2 which includes metadata MT including information which shows thatit is a full circumference panorama image, and a panorama image filePDr1 which includes panorama image data PD, and associates these by thefile link information LK1 and LK2.

On the other hand, the display image generation section 20 in the imageprocessing apparatus 2 acquires the metadata MT associated with thepanorama image data PD by using file link information, and judgeswhether or not the panorama image data PD is a full circumferencepanorama image from the metadata MT. Then, display image data isgenerated by performing a display projection process at the time whenthe panorama image data PD is judged to be a full circumference panoramaimage, and a projection display is executed.

By such a configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

Note that, in FIG. 15 and FIG. 16 it is described by examples usingmetadata MT, it can also be considered by using the display processingprogram PG stated in the second configuration example.

For example, a display processing program PG which is not metadata MT isincluded in the data file PDr2 shown in FIG. 15B. It is an example inwhich, if a display processing program PG is included in the associateddata file PDr2, the display image generation section 20 performs ageneration process of display image data PDdsp for a projection displayby using this display processing program PG.

Information which shows that it is a full circumference panorama imageis included as a different file with the above described panorama imagedata PD, and modes in which these are associated can be considered forvarious types. As a related technique 1, for example, a Unique MaterialIdentifier (UMID) is well known as information which refers to videomaterial in video editing data (for example, refer to JP 2004-312233A).For example, the data file PDr2 of FIG. 15B may associate metadata MTwith panorama image data PD in the panorama image file PDr1 which is adifferent file, by a mode such as UMID.

<6. Fourth Configuration Example (Dedicated Format)>

A fourth configuration example of an embodiment will be described. Inthe fourth embodiment, the association section 11 of the imageprocessing apparatus 1 sets panorama image data itself to data of aspecific format, as information which shows that it is a fullcircumference panorama image.

Further, the display image generation section 20 of the image processingapparatus 2 judges that it is a full circumference panorama image, bysetting panorama image data to data of a specific format, and generatesdisplay image data by performing a display projection process.

FIG. 17A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this fourthconfiguration example, the association section 11 has a formatprocessing section 11 f.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The format processing section 11 f judges whether or not the panoramaimage data PD is a full circumference panorama image, by any of panoramamode type information, 360° viewing angle judgment information and bothend alignment information in the panorama processing information Ip.Then, in the case of a full circumference panorama image, a panoramaimage file PDr, which includes panorama image data PD of a dedicatedformat, is generated. In the case where it is not a full circumferencepanorama image, a panorama image file PDr, which includes panorama imagedata PD of a general normal format, is generated.

Note that, for example, panorama image data PD of a normal format is animage data form usually used such as a Joint Photographic Experts Group(JPEG) format or the like. On the other hand, a dedicated format is setto a new data format exclusively for a full circumference panoramaimage.

FIG. 17B shows the display image generation section 20 in the imageprocessing apparatus 2. In this fourth configuration example, thedisplay image generation section 20 has a format judgment section 20 g,and an image generation processing section 20 c.

A panorama image file PDr, which is reproduced from a recording mediumor is received, is input to the format judgment section 20 g. The formatjudgment section 20 g judges whether panorama image data PD included inthe panorama image file PDr is a dedicated format or a general format.Then, the extracted panorama image data PD and format judgmentinformation FM are supplied to the image generation processing section20 c.

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is judged whether or not it is a full circumference panoramaimage by the format judgment information FM, and generation of displayimage data PDdsp for a projection display is performed if it is a fullcircumference panorama image. In the case where the panorama image dataPD is not a full circumference panorama image, a process for anotherdisplay, for example, a scroll display or a single view display, isperformed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 18.

FIG. 18A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F41, the panorama combination section 10 captures an input imagegroup FMS.

In step F42, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in the panorama combination process processing, with thepanorama image data PD, to the association section 11 (the formatprocessing section 11 f).

In step F43, the association section 11 (the format processing section11 f) confirms the panorama processing information Ip, and performs ajudgment of whether or not the generated panorama image data PD is afull circumference panorama image. Similar to that stated in the firstconfiguration example, the judgment of whether or not it is a fullcircumference panorama image can be considered by a system which refersto 360° viewing angle judgment information in the panorama processinginformation Ip, a system which refers to both end alignment information,a system which refers to panorama mode type information or the like.

In the case where it is a full circumference panorama image, it proceedsfrom step F44 to F45, and the association section 11 (the formatprocessing section 11 f) generates a panorama image file PDr byconverting the panorama image data PD into a dedicated format.

In the case where it is not a full circumference panorama image, itproceeds from step F44 to F46, and the association section 11 (theformat processing section 11 f) generates a panorama image file PDrwhich includes panorama image data PD of a normal format.

In step F47, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

FIG. 18B shows the processes of the display image generation section 20of the image processing apparatus 2.

In step F81, the display image generation section 20 (the formatjudgment section 20 g) captures a panorama image file PDr reproduced bya recording reproduction section which is not illustrated, or receivedby a communication section which is not illustrated.

In step F82, the display image generation section 20 (the formatjudgment section 20 g) judges the format of panorama image data PDincluded in the panorama image file PDr.

Then, in step F83, the display image generation section 20 branches theprocesses in accordance with a format judgment result.

In the case where it is judged to be a dedicated format, the displayimage generation section 20 (the image generation processing section 20c) proceeds to step F84, display image data PDdsp is generated byperforming a display projection, and in step F85, this display imagedata PDdsp is output. The processes of these steps F84 and F85 arecontinued up to it becomes a display end, in step F86, by a useroperation or an automatic process. In this way, in the display section,for example, a display so as to look over the full circumference, adisplay of an arbitrary direction or the like is executed as aprojection display.

Note that, the viewing angle of the panorama image data PD, which isused by a projection calculation in the display projection process, maybe set to 360°.

Further, information of a projection surface of the panorama image dataPD, horizontal and vertical viewing angle information for scrolllimitation or the like may be considered by being embedded in the dataform of a dedicated format.

In step F83, in the case where it is judged that it is not a dedicatedformat, the display image generation section 20 (the image generationprocessing section 20 c) generates, in step F87, display image dataPDdsp as another system, for example, a scroll display or a single viewdisplay. Then, in step F88, a process is performed which outputs thedisplay image data PDdsp. The processes of these steps F87 and F88 arecontinued up to it is judged to be a display end, in step F89. In thiscase, in the display section, a scroll display or a single view displayother than a projection display is executed.

In the fourth configuration example such as described above, in theimage processing apparatus 1, the association section 11 makes adedicated format in the case of a full circumference panorama image.

On the other hand, in the image processing apparatus 2, by judging thatthe panorama image data PD is a full circumference panorama image if itis a dedicated format, the display image generation section 20 generatesdisplay image data PDdsp for a projection display by performing adisplay projection process.

By this configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be performed, and a display which has a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display method, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

<7. Fifth Configuration Example (Application Example to an ImagingApparatus)> (7-1: Configuration of the Imaging Apparatus)

To continue, as a fifth configuration example of an embodiment, animaging apparatus 50 will be described as a specific example by a devicewhich includes the image processing apparatuses 1 and 2 of the presentdisclosure.

FIG. 19 shows a configuration example of the imaging apparatus 50.

The imaging apparatus 50 includes a lens unit 100, an imaging element101, an image processing section 102, a control section 103, a displaysection 104, a memory section 105, a recording device 106, an operationsection 107, a sensor section 108, and a communication section 109.

The lens unit 100 collects an optical image of a photographic subject.The lens unit 100 has a function which adjusts a focal distance, aphotographic subject distance, an aperture or the like, so as to obtainan appropriate image, in accordance with an instruction from the controlsection 103. Further, it also has a camera-shake correction function foroptically suppressing blurring of an image.

The imaging element 101 optically converts the optical image collectedby the lens unit 100, and converts it into electrical signals.Specifically, it is implemented by a Charge Coupled Device (CCD) imagesensor, a Complementary Metal Oxide Semiconductor (CMOS) image sensor orthe like.

The image processing section 102 is constituted from a sampling circuitwhich samples electrical signals from the imaging element 101, an A/Dconversion circuit which converts analog signals into digital signals,an image processing circuit which applies a prescribed image process todigital signals or the like. Here, this image processing section 102 isshown as performing a process which obtains frame image data by imagingwith the imaging element 101, and performing a process which combines apanorama image.

This image processing section 102 includes not only a dedicated hardwarecircuit, but also a Central Processing Unit (CPU) or a Digital SignalProcessor (DSP), and can perform software processes to accommodateflexible image processes.

The control section 103 is constituted from a CPU and a control program,and performs a control of each section of the imaging apparatus 50. Thecontrol program itself is actually stored in the memory section 105, andis executed by the CPU.

The processes of the panorama combination section 10 and the associationsection 11 disclosed in each of the above described configurationexamples, and additionally the processes of the display image generationsection 20, are executed by the control section 103 and the imageprocessing section 102. Therefore, the image processing apparatuses 1and 2 of the present disclosure are implemented by the control section103 and the image processing section 102.

The display section 104 is constituted from a D/A conversion circuitwhich makes image data processed by the image processing section 102 andstored in the memory section 105 analog, a video encoder which encodesthe image signals made analog in video signals of a form which isappropriate for a display device of a later stage, and a display devicewhich displays an image corresponding to the input video signals.

The display device is implemented, for example, by a Liquid CrystalDisplay (LCD), an organic Electroluminescence (EL) panel or the like,and also has a function as a finder.

The memory section 105 is constituted from a semiconductor memory suchas a Dynamic Random Access Memory (DRAM), and image data processed bythe image processing section 102, control programs and various types ofdata in the control section 103 or the like are temporarily recorded.

The recording device 106 is constituted by a recording medium such as asemiconductor memory such as a flash memory (Flash Memory), a magneticdisc, an optical disc or a magneto-optical disc, and a recording andreproduction circuit/mechanism for these recording mediums.

At the time of imaging by the imaging apparatus 50, encoding isperformed in a Joint Photographic Experts Group (JPEG) form by the imageprocessing section 102, and JPEG image data stored in the memory section105 is recorded to a recording media.

At the time of reproduction, JPEG image data held in a recording mediumis read to the memory section 105, and a decoding process is performedby the image processing section 102. It is possible for the decodedimage data to be displayed by the display section 104, or to betransmitted and output to an external device by the communicationsection 109.

The operation section 107 includes input devices such as hardware keyssuch as a shutter button, up, down, left and right arrow keys, adetermination key and a cancel key, an operation dial, a touch panel anda zoom lever, detects an input operation of a photographer (user), andtransmits it to a control section 103. The control section 103determines an operation of the imaging apparatus 50 in accordance withan input operation of the user, and performs a control so that eachsection performs necessary operations.

The sensor section 108 is constituted by a gyro sensor, an accelerationsensor, a geomagnetic sensor, a Global Positioning System (GPS) sensoror the like, and performs detection of various types of information.Apart from being added as metadata for captured image data, thisinformation is also used for various types of image processes andcontrol processes.

The communication section 109 performs communication with an externaldevice or network communication. Specifically, various types of examplescan be considered for the communication section 109, such as a USBcommunication section, a LAN communication section, a public networkcommunication section such as the internet, or a dedicated communicationsection with a specific device. Further, the form of a transmission linemay be either a wireless transmission line by electronic waves orinfrared rays, or a cable transmission line by a cable connection. Thesignal mode can be assumed to be digital electronic signalcommunication, analog electronic signal communication, opticalcommunication or the like.

The image processing section 102, the control section 103, the displaysection 104, the memory section 105, the recording device 106, theoperation section 107, the sensor section 108 and the communicationsection 109 are mutually connected via a bus 110, and image data,control signals or the like are exchanged.

(7-2: Panorama Combination and Association Processes)

Next, a panorama combination process and an association process of theimaging apparatus 50 of the present embodiment will be described indetail.

FIG. 20 shows, as functional configurations, the processes executed inthe image processing section 102 and the control section 103 for thepanorama combination process and the association process, and shows theprocesses executed by these functional configuration portions.

Note that, here, the example using metadata stated in the abovedescribed first configuration example will be described as theassociation process.

As shown by the dashed line in FIG. 20, the panorama combination section10 and the association section 11 are included as functionalconfigurations. That is, the image processing apparatus 1 is formedinside the imaging apparatus 50, by having a function included as theassociation section 11 by at least the image processing section 102, thecontrol section 103 or the like.

Further, the process functions executed by the functions as the panoramacombination section 10 will be described. In FIG. 20, the processes(algorithm flow) executed as the panorama combination section 10 areshown by each block. That is, in the panorama combination section 10, apre-process 200, an image registration process 201, a movingphotographic subject detection process 202, a detection/recognitionprocess 203, a 360° imaging judgment process 204, a 360° optimizationprocess 205, a combination projection process 206, a seam determinationprocess 207 and a stitch process 208 are performed.

Each of the processes will be described.

As described in FIG. 1, an input image group FMS which becomes a targetof the pre-process 200 is frame image data FM#0, FM#1, FM#2 . . .sequentially obtained at the time when a photographer executes panoramaimaging by the imaging apparatus 50.

In the panorama combination section 10, first, the pre-process 200 forpanorama combination is performed for images (each frame image data)captured by a panorama imaging operation of a photographer. Note that,here, an image process the same as that at the time of normal imaging isperformed for the frame image data FM.

The input image is influenced by aberrations based on thecharacteristics of the lens unit 100. In particular, distortionaberrations of the lens will negatively affect the image registrationprocess 201, and will reduce the accuracy of alignment. In addition,since artifacts are also generated in the seam vicinity of a combinedpanorama image, correction of the distortion aberrations is performed inthis pre-process 200. By the correction of the distortion aberrations,there will be an effect in which the accuracy of the moving photographicsubject detection process 202 and the detection/recognition process 203are improved.

Next, in the panorama combination section 10, the image registrationprocess 201, the moving photographic subject detection process 202 andthe detection/recognition process 203 are performed for the frame imagedata to which the pre-process 200 has been performed.

It may be necessary to convert a plurality of frame image data intocoordinates in a simple coordinate system at the time of panoramacombination, and this simple coordinate system will be called a panoramacoordinate system.

The image registration process 201 is a process which inputs twosuccessive frame image data, and performs alignment in the panoramacoordinate system. While information obtained by the image registrationprocess 201 for two frame image data is a relative relation between atthe two image coordinates, a coordinate system of all the frame imagedata can be converted into a panorama coordinate system, by selectingone out of the plurality of image coordinate systems (for example, thecoordinate system of the initial frame image data), and fixing it to thepanorama coordinate system.

The specific processes performed in the image registration process 201will be roughly divided into two as follows.

1. Detecting local movements within an image.

2. Obtaining full image global movements from the above describedobtained local movement information.

In the above described process 1, generally:

-   -   Block matching    -   Characteristic point extraction and characteristic point        matching such as Harris, Hessian, SIFT, SURF or FAST        or the like are used, and local vectors of characteristic points        of the images are obtained.

In the above described process 2, robust estimation techniques such as:

-   -   A Least Squares Method    -   An M-Estimator    -   A Least Median Squares (LMedS) method    -   A RANdom SAmple Consensus (RANSAC)        is used as a local vector group obtained in the above described        process 1, and an optimal affine transformation matrix or a        projection transformation matrix (Homography), which states a        relation between 2 coordinate systems, is obtained. In the        present disclosure, this information will be called image        registration information.

Further, in the panorama composition process, a part of a movingphotographic subject in which a moving photographic subject is presentin an imaging scene is divided in terms of the characteristics in whicha plurality of frame image data is combined, and will cause a breakdownof an image such as becoming blurry or a reduction of the image quality.Accordingly, it is preferable to determine the seam of a panorama, upondetecting a moving photographic subject, by avoiding the movingphotographic subject.

The moving photographic subject detection process 202 is a process whichinputs 2 or more successive frame image data, and performs detection ofa moving photographic subject. In the case where an example of aspecific process is at a threshold or above where there is a differencein values of the pixels of the 2 frame image data in which alignment isactually performed, by image registration information obtained by theimage registration process 201, these pixels will be judged to be amoving photographic subject.

Alternatively, a judgment may be performed by using characteristic pointinformation, which is judged as outliers (outliers) at the time ofrobust estimation of the image registration process 201.

In the detection/recognition process 203, position information such as ahuman face or body or an animal within captured frame image data isdetected. Even if there is a high possibility of a human or animal beinga photographic subject, which is not moving, in the case where the seamof the panorama is determined on this photographic subject, a sense ofincompatibility will often occur visually compared to other objects, andtherefore it is preferable to determine a seam by avoiding theseobjects. That is, information obtained in this detection/recognitionprocess 203 is used for supplementing information of the movingphotographic subject detection process 202.

A seam (junction) is determined in the seam determination process 207,which will be described later, based on information obtained in themoving photographic subject detection process 202 and thedetection/recognition process 203.

The 360° imaging judgment process 204 and the 360° optimization process205 perform processes specialized for 360° full circumference imaging.

Here, in order to understand these processes, various problems at thetime of 360° imaging will be described.

FIG. 21A, FIG. 21B and FIG. 22 are examples of panorama images in eachof the cases. Note that, in these figures, in order to confirmcontinuality of the left end and the right end of the panorama images, apart of the left end regions of the panorama images are shown copied(image RCP) onto each of the right ends of the panorama images (theright side of the boundary line BD). An image portion in which the imageRCP is excluded, that is, a portion shown by the range RP of the outputpanorama image, is a final one panorama image.

In the case where it is a 360° full circumference panorama, it isdesirable for an image having continuality without gaps to be naturallyconnected on both sides of this boundary line BD.

In a panorama image in which a plurality of still images are combined,it will be difficult to accurately know a general panorama viewingangle. Usually, a focal distance (f: pixel conversion value) of the lensof the imaging apparatus, a pixel number (1) of a sweep direction of thepanorama image, and a panorama viewing angle (θ) have the followingrelation.

1=fθ

However, since there are actually influences such as:

-   -   A radius of gyration when sweeping    -   Correction errors of distortion aberrations    -   Errors of a registration process, there can be significant        differences with the viewing angle of an actual scene.

In the case where the registration process is limited to only parallelmovement, and simplification is performed on an algorithm such asomitting a projection process, for the processing performance of theimaging apparatus, these errors will become more pronounced.

FIG. 21A is an example of a panorama image, to which a suitable 360°panorama image number is determined from the above equation, and acomposition process is performed. It is combined by significantlyexceeding 360° for errors, and there is no continuality of the left endand the right end (boundary line BD portion) of the panorama image.

In this way, while it is difficult to accurately determine a generalpanorama viewing angle, in the case where it is a 360° fullcircumference image, it is possible to determine a viewing angle byconsidering a constraint condition in which continuity of the left endand the right end of a panorama is obtained.

FIG. 23 will be referred to. FIG. 23 is a schematic diagram of an imagegroup (frame image data FM#0 to FM#(n−1)) obtained by 360° panoramaimaging.

Specifically, the following conditions are established, at the time whenthe sweep direction is assumed to be from left to right, and the firstframe image data FM#0 used on the panorama left side is set to a 0°position, for the frame image data FM#0A of an image entirely the sameas this, which is at a position prior to 360°, and the final frame imagedata FM#(n−1) used on the right end of the panorama image.

-   -   FM#0A and FM#(n−1) have overlapping regions.    -   FM#(n−1) is an image nearer to FM#0A on the left side than        FM#0A.        By retrieving the frame image data FM#(n−1) as a final image        which satisfies these conditions, a 360° position can be fixed.

In the imaging apparatus 50 of this example, this process is performedby the 360° imaging judgment process 204.

In the 360° imaging judgment process 204, in the case where the finalframe image data FM#(n−1) which satisfies the above described twoconditions can be determined, it is judged that the panorama image datawithin the combination process is a full circumference panorama image by360° imaging. On the other hand, in the case where the final frame imagedata FM#(n−1) which satisfies the above described two conditions is notable to be determined, it is judged that the panorama image data withinthe combination process is not a full circumference panorama image.

This judgment result becomes 360° viewing angle judgment informationwhich is one of the above described panorama processing information Ip.In the 360° imaging judgment process 204, a process is performed whichoutputs the 360° viewing angle judgment information to the associationsection 11.

Note that, this 360° imaging judgment process 204 may be performed inthe case where panorama imaging is executed in a 360° panorama imagingmode. In the cases other than a 360° panorama imaging mode (for example,a 180° panorama imaging mode or the like), it may not be necessary toperformed this 360° imaging judgment process 204. Note that, in thiscase, it may be set to information which shows it is a non-fullcircumference panorama image as the 360° viewing angle judgmentinformation.

Further, in the process of this 360° imaging judgment process 204, itmay be output to the association section 11 by generating horizontal andvertical viewing angle information and panorama mode type information.

FIG. 21B is an example of a result of combining the final imagedetermined in the 360° imaging judgment process 204, that is, up to theframe image data FM#(n−1) which conforms to the above described twoconditions.

Since a 360° position is determined in the 360° imaging judgment process204, it will become a 360° image as a panorama image. However, thisexample does not have continuality of the left end and the right end ofa panorama similar to that of the example of FIG. 21A. While theposition of the horizontal direction is suitable in this example, theposition of the vertical direction will shift.

Since this performs alignment of each image between adjacent images, itwill be generated by having errors accumulated in the registrationprocess.

That is, in FIG. 23, there is a phenomenon in which a position (aposition shown as the frame image data FM#0B) to be a first image priorto 360° for becoming a full circumference panorama with continuality iscalculated for a position shown as the frame image data FM#0A for anaccumulated error of the registration process.

In order to address this phenomenon, in the following 360° optimizationprocess 205, in the case where it is judged to be 360° fullcircumference imaging in the 360° imaging judgment process 204, theimage registration information is optimized, by using the imageregistration information input from the image registration process 201,and the final image information determined by the 360° imaging judgmentprocess 204, and is appropriately corrected as a 360° full circumferencepanorama.

This process is the previously stated process of both end alignment.That is, image registration information is adjusted so that there are noimage gaps in the vertical direction and the horizontal direction, atboth ends of the panorama image which are the 0° and 360° positions ofthe panorama image.

When described in FIG. 23, the positions of all the frame image dataFM#0 to FM#(n−1) become optimized and corrected, so that the position ofthe frame image data FM#0A matches the position of the frame image dataFM#0B.

Specifically, there is a technique which simply distributes accumulatederrors to all the image registration information, a technique whichperforms optimization without impairment as long as the originalregistration information can be made using a least squares method or thelike.

For example, a technique which simply distributes is that which isstated as follows. There are errors ΔX of the horizontal direction anderrors ΔY of the vertical direction such as shown in FIG. 23 at theposition of the frame image data FM#0A and the position of the frameimage data FM#0B. The x-coordinate positions and the y-coordinatepositions of n frame image data FM#0 to FM#(n−1) constituting thepanorama image are each respectively shifted ΔX/n and ΔY/n. In this way,errors will be absorbed little by little in each frame image data FM,and the positions of virtual frame image data FM#0A as an image the sameas that of the first frame image data FM#0 will match the positionsshown as the frame image data FM#0B.

While each of the frame image data FM#0 to FM#(n−1) will shift slightlymore than the original coordinates, the entire image quality will havehardly any deterioration.

FIG. 22 is an example of a result in which panorama combination isperformed by using image registration information corrected by having aboth end alignment performed such as described above in the 360°optimization process 205. By the 360° imaging judgment process 204 andthe 360° optimization process 205, it can be confirmed that an imagewill be naturally connected, without vertical and horizontal gaps atboth ends of a panorama, as the range RP of an output panorama image.

By the 360° optimization process 205 such as described above,information of whether or not a both end alignment is executed becomesboth end alignment information which is one of the panorama processinginformation Ip.

In the 360° optimization process 205, a process is also performed whichoutputs the both end alignment information to the association section11.

Next, in the combination projection process 204, a projection process isperformed on a single plane surface or a single curved surface such as acylindrical surface or spherical surface for all the frame image data,based on the image registration information obtained by the imageregistration process 201 (or image registration information corrected inthe 360° optimization process 205). Further, at the same time, aprojection process is performed on the same plane surface or curvedsurface for moving photographic subject information anddetection/confirmation information. A projection surface may beautomatically selected from the viewing angle or the like at the time ofimaging, or may be set in accordance with a user operation.

By considering optimization of pixel processing, the combinationprojection process 204 of frame image data may be performed as a processpreceding the stitch process 208, or as a part of the stitch process208. Further, for example, it may be simply performed as a part of thepre-process 200, prior to the image registration process 201. Further,by being additionally simplified, it may be handled as an approximationof a cylindrical projection process without performing this processing.

Note that, the type of information of a projection surface in thiscombination projection process 206 becomes combination projectionsurface information which is one of the panorama processing informationIp.

In the combination projection process 206, a process is also performedwhich outputs the combination projection surface information to theassociation section 11.

The seam determination process 207 is a process which sets, to an input,image data from the combination projection process 206, imageregistration information from the image registration process 201, movingphotographic subject information from the moving photographic subjectdetection process 202 and detection/confirmation information from thedetection/recognition process 203, and determinates appropriate seams SM(seams SM0 to SM(n−2) described in FIG. 2 and FIG. 3), which have littlefailure as a panorama image.

In the seam determination process 207, first, a cost function is definedfor overlapping regions between adjacent frame image data from the inputinformation.

For example, a total value may be set to a function value, whichassociates each appropriate overlap with the moving photographic subjectinformation from the moving photographic subject detection process 202and detection/confirmation information from the detection/recognitionprocess 203, for each pixel of the overlapping regions.

In this case, since it has the meaning of the cost function valueincreasing with an increase in objects such as moving photographicsubjects being present at this point, a collection of points with lowcost function values may be seamed, in order for failures in thepanorama image to be suppressed to a minimum.

In the case where there are n images (frame image data) used forpanorama image combination, the number of these overlapping regions willbecome n−1, and the cost functions will be defined as n−1. Therefore, inorder for optimal seams to be selected as an entire panorama image, acombination is obtained in which these n−1 cost functions are minimized.This is generally called a combination optimization problem, and thefollowing solutions are well-known.

-   -   Methods for obtaining an exact solution        -   Branch and bound method        -   Memorization        -   Dynamic Programming technique (Dynamic Programming)        -   Graph cut    -   Methods which obtain an approximate solution        -   Local search method (Hill-climbing method)        -   Annealing method (Simulated Annealing)        -   Taboo search        -   Generic algorithm (Generic algorithm)

All of the seams SM1 to SM(n−2) can be obtained by any of the abovedescribed methods.

In the stitch process 208, a final panorama image is combined, by usingall of the seams SM1 to SM(n−2) determined such as described above, andeach of the frame image data FM#0 to FM#(n−1).

A blend process for reducing unnaturalness of junctions is performed forthe regions of the seam surroundings, copying of simple pixels, that is,re-sampling to a panorama coordinate system, is only performed for theregions other than these, and the entire image is connected.

By finally sweeping by considering a camera shake amount and trimmingunnecessary portions of the vertical direction, a panorama image(panorama image data PD) with a wide viewing angle can be obtained, suchas that which has a sweep direction set to a long side direction.

The above become the processes of the panorama combination section 10.

In the case of the present embodiment, the processes of the associationsection 11 are additionally performed for the panorama image data PDgenerated by the panorama combination section 10.

In the functional configuration example of FIG. 20 in the imagingapparatus 50 of this example, the previous functional configuration ofFIG. 11A is adopted as the first configuration example. The associationsection 11 performs a metadata generation process 301 and a dataintegration process 302.

Various types of panorama processing information Ip are supplied fromthe panorama combination section 10 to the association section 11. Thatis, 360° viewing angle determination information obtained in the 360°imaging judgment process 204, both end alignment information obtained inthe 360° optimization process 205, and combination projection surfaceinformation obtained in the combination projection process 206 aresupplied to the association section 11.

Note that, panorama mode type information and horizontal and verticalviewing angle information may also be additionally supplied.

The association section 11 generates metadata MT reflecting the panoramaprocessing information Ip as a metadata generation process 301.

Note that, since a judgment of whether or not it is a full circumferencepanorama image is performed in the processes of the display imagegeneration section 20, which will be at least described later, and fromthe viewpoint of being used for performing a display projection process,it may not be necessary for the metadata MT to include all of the 360°viewing angle determination information, both end alignment information,combination projection surface information, panorama mode typeinformation and horizontal and vertical viewing angle information.

While it has been described in the first configuration example to thefourth configuration example, the judgment of whether or not it is afull circumference panorama image by the display image generationsection 20 may refer to any of the 360° viewing angle determinationinformation, both end alignment information and panorama mode typeinformation. However, combination projection surface information may benecessary for appropriately performing a display projection process.

Accordingly, at least one of:

-   -   360° viewing angle information and combination projection        surface information    -   Both end alignment information and combination projection        surface information    -   Panorama mode type information and combination projection        surface information        may be included as the panorama processing information Ip,        generated by the panorama combination section 10, and made into        metadata by the metadata generation section 11 a.

In the data integration process 302, the association section 11generates a panorama image file PDr by integrating the panorama imagedata PD generated by the panorama combination section 10 with themetadata MT generated by the metadata generation process 301.

For example, the panorama image file PDr becomes a structure which hasimage data and metadata such as that disclosed in FIG. 11B. Morespecifically, an example can be considered in which metadata such as anExchangeable Image File Format (EXIF) is embedded within the image data.Note that, it may be set to a file in which the panorama image data PDand the metadata MT are independent.

In the imaging apparatus 50 of the present example, processes areperformed as the above described panorama combination section 10 andassociation section 11 in the image processing section 102 and thecontrol section 103, and as a result of this, the panorama image filePDr is recorded to the recording device 106.

Note that, the panorama image file PDr may be externally transmittedfrom the communication section 109.

An example of the series of processes from panorama imaging up torecording, which includes the processes of the above described panoramacombination section 10 and association section 11, is shown in FIG. 24.

The image capturing of step F100 captures one still image in a panoramaimaging mode, and means a capturing process as one frame image datawithin the imaging apparatus 1. That is, imaging signals obtained by theimaging element section 101 are imaging signal processed by the imageprocessing section 102, by a control of the control section 102, andbecome one frame image data.

This frame image data may be supplied to the panorama combinationprocesses (the processes of the panorama combination section 10 of FIG.20) by the image processing section 102 as it is, or it may be suppliedto the panorama combination processes by the image processing section102 as one frame image data after being embedded once in the memorysection 105.

The processes from step F101 onwards are performed, in accordance withan input of frame image data based on step F100, by the panoramacombination section 10 of FIG. 20 implemented by the image processingsection 102 and the control section 103.

In step F101, the panorama combination section 10 performs thepre-process 200.

In step F102, the panorama combination section 10 performs the imageregistration process 201.

In step F103, the panorama combination section 10 performs the movingphotographic subject detection process 202.

In step F104, the panorama combination section 10 performs thedetection/recognition process 203.

Note that, pixel information, image registration information, movingphotographic subject information, detection/confirmation information orthe like of an image obtained in each of these processes are temporarilystored in the memory section 105 as information corresponding to frameimage data of respective process targets.

The above described processes are performed for each frame image data,up to it is an imaging end, in step F105.

If the above described processes are finished for all the captured frameimage data FM, in step F106, the panorama combination section 10performs the 360° imaging judgment process 204.

In step F107, the panorama combination section 10 performs the 360°optimization process 205.

Then, in step F108, the panorama combination section 10 performs thecombination projection process 206, by using image registrationinformation or the like adjusted in the 360° optimization process 205.

In step F109, the panorama combination section 10 determines the seamsSM0 to SM(n−2), by performing the seam determination process 207.

In step F110, the panorama combination section 10 executes the stitchprocess 208. That is, each frame image data is connected by each of theseams SM0 to SM(n−2). A blend process is also performed when beingconnected. Hereinafter, one panorama image data PD is generated such asthat shown in FIG. 3A.

To continue, the association section 11 performs the metadata generationprocess 301. That is, metadata MT is generated which reflects thepanorama processing information IP supplied from the panoramacombination section 10.

Then, in step F112, the association section 11 generates a panoramaimage file PDr by performing the data integration process 302.

The control section 103 performs a control which causes the panoramaimage file PDr generated in such a process as the panorama combinationsection 10 and association section 11 to be recorded to the recordingdevice 106.

In the above described processes, the panorama image file PDr whichincludes the panorama image data PD and the metadata MT is recorded, andreproduction and display or the like becomes possible after this.

(7-3: Panorama Image Display)

To continue, an operation will be described in the case where therecorded panorama image file PDr such as described above is selected anddisplayed in the imaging apparatus 50.

First, the display operation modes performed by the display section 104of the imaging apparatus 50 will be described in FIG. 25.

In the imaging apparatus 50, display operations of the above describedfour display modes of a single view display, a list display, a scrolldisplay and a projection display are capable of being executed. FIG. 25shows state transitions of the four display modes.

In order for simplification of the description, only a plurality ofpanorama image files PDr will be recorded to the recording device 106.

First, when a user specifies reproduction and display by using keys orthe like of the operation section 107, the control section 103 sets theimaging apparatus 50 to a reproduction and display state. At this time,it transitions from an initial state to a single view display mode.Here, for example, the latest image of the panorama image file PDr isdisplayed such as that of FIG. 7A, for example. In this single viewdisplay mode, for example, it can be switched other images of thepanorama image file PDr by having the user press the right arrow key andthe left arrow key of the operation section 107.

Further, in a single view display, the state transitions to a listdisplay such as that of FIG. 7B when the upper arrow key is pressed, anda list view of images is performed.

The state transitions again to a single view display, by having the userselect an image to be a target using the up, down, left and right arrowkeys, on this list display screen, and press the determination key, andthe selected image of the panorama image file PDr is displayed.

In the case where the determination key is pressed, for example, in asingle view display, an enlarged display is performed. Note that, here,an enlarged display will be a scroll display or a projection display.

The control section 103 refers to the metadata of the displayed image ofthe panorama image file PDr, and in the case where the panorama imagefile PDr is a full circumference panorama image, it transitions thestate to a projection display mode, and a projection display isexecuted. In this case, a projection process is performed, in accordancewith projection surface information at the time of combining which isrecorded as metadata, and a projection display is started.

In this projection display mode, it is possible for a user to specify achange of viewing direction by using the up, down, left and right arrowkeys, or a zoom-in, zoom-out by using the zoom lever. Further, in thiscase, by having horizontal and vertical viewing angle informationrecorded in metadata for a viewing direction and zoom value, alimitation is performed by the horizontal viewing angle or the verticalviewing angle as a panorama image, and therefore will not refer to thatoutside of the image.

In the case where there is no key operation for a fixed time, it ispossible for an automatic viewing change to be performed by a menusetting.

In this projection display mode, it transitions to a single view displaystate when the user presses a cancel key, and a single view display isperformed again.

In the case where the determination key is pressed in a single viewdisplay, and the displayed image of the panorama image file PDr is not afull circumference panorama image, it transitions to a scroll displaymode, and a scroll display is executed.

In a scroll display mode, while scrolling is automatically started inthe long side direction of a panorama, it is possible for the user toalso specify scrolling in an arbitrary direction by using the up, down,left and right arrow keys, or a zoom-in or zoom-out by using the zoomlever.

In a scroll display mode, it transitions to a single view display statewhen the user presses a cancel key, and a single view display isperformed again.

In the imaging apparatus 50 of the present embodiment with such displaymode transitions as described above, a projection display is performedif it is a full circumference panorama image, and automatic switching ofdisplay operations, such as a scroll display being performed if it isnot a full circumference panorama image, is performed.

If it is a full circumference panorama image, a high-quality projectiondisplay is automatically provided for the user. On the other hand, inthe case where it is not a full circumference panorama image, it ispossible for image distortions or the like to occur when performing aprojection display, and a high-quality display is not be able to beguaranteed. Accordingly, a display of a panorama image is performed in ascroll display. That is, an appropriate display mode is selected inaccordance with the panorama image.

Configurations and processes for performing such displays will bedescribed in FIG. 26 and FIG. 27.

FIG. 26 shows the processes executed in the display image generationsection 20 included in the control section 103 for the generationprocess of display image data PDdsp.

Note that, an example based on the example using metadata stated in theabove described first configuration example will be set as the processesof the display image generation section 20.

The display image generation section 20 executed by software in thecontrol section 103 performs a data separation process 401, a metadatainterpretation process 402 and an image generation process 403 such asillustrated.

The display image generation section 20 separates metadata MT andpanorama image data PD, as the data separation process 401, for apanorama image file PDr of a display target read from the recordingdevice 106.

The separated metadata MT is interpreted by the metadata interpretationprocess 402, and is supplied to the image generation process 403 aspanorama processing information Ip (panorama mode type information, 360°viewing angle judgment information, both end alignment information,combination projection surface information, horizontal and verticalviewing angle information).

In the image generation process 403, display image data PDdsp isgenerated in accordance with display mode information Md, and is outputto the display section 104.

A display system selection process 403 a, a single view display process403 b, a list display process 403 c, a scroll display process 403 d anda projection display process 403 e are performed as the image generationprocess 403.

In the display system selection process 403 a, a process is performedwhich selects what display is to be executed, based on the display modeinformation Md corresponding to the above described display modetransitions based on a user operation or the like. Further, when movingto a scroll display mode or a projection display mode at the time of astate of a single view display mode, a scroll display mode and aprojection display mode are selected, by referring to the panoramaprocessing information Ip interpreted in the metadata interpretationprocess 402.

In the single view display process 403 b, display image data PDdsp isgenerated as a single view display for the supplied panorama image dataPD.

In the list display process 403 b, for example, a list image isgenerated from thumbnail images or the like of each panorama image filePDr read from the recording device 106, and a process is performed whichsets this to display image data PDdsp as a list display.

In the scroll display process 403 d, display image data PDdsp isgenerated as a scroll display for the supplied panorama image data PD.

In the projection display process 403 e, a display projection process isperformed for the supplied panorama image data PD, and display imagedata PDdsp is generated as a projection display.

The display image data PDdsp generated in any of these processes issupplied to the display section 104 and a screen display is performed.

FIG. 27 shows a process example of the control section 103 (the displayimage generation section 20) corresponding to the display modetransitions of FIG. 25.

When reproduction and display is stated in accordance with a useroperation or the like, in step F201, the display image generationsection 20 performs the processes of a single view display mode. Forexample, the display image generation section 20 reads the latestpanorama image file PDr from the recording device 106, generates displayimage data PDdsp by performing the single view display process 403 b,and supplies it to the display section 104.

In the period of a single view display mode, the display imagegeneration section 20 monitors a trigger of a mode transition or adisplay end in steps F202, F203 and F206 (the display system selectionprocess 403 a).

If the display image generation section 20 detects an end trigger, thereproduction and display operation ends from step F202.

The display image generation section 20 proceeds from step F203 to F204when an operation of a list display mode is detected, and the listdisplay process 403 c is performed. For example, a list image isgenerated by reading thumbnail images of each panorama image file PDrfrom the recording device 106, and is supplied to the display section104 as display image data PDdsp.

In step F205, the display image generation section 20 monitors a triggerof a list display end (the display system selection process 403 a). Inthe case of a list display end, the single view display process 403 b isperformed by returning to step F201. For example, display image dataPDdsp is generated as a single view display, for a panorama imageselected in the list, and is supplied to the display section 104.

When an enlarged display (a scroll display or a projection display) isspecified in a single view display state, the display image generationsection 20 proceeds from step F206 to F207, and first confirms thepanorama processing information Ip. That is, it is judged whether or notthe image presently in a single view display is a full circumferencepanorama image. In this case, the display image generation section 20can judge whether or not the panorama image data PD is a fullcircumference panorama image, by any of panorama mode type information,360° viewing angle judgment information and both end alignmentinformation in the panorama processing information Ip.

In the case where it is a full circumference panorama image, the displayimage generation section 20 proceeds from step F208 to F211, generatesdisplay image data PDdsp, by performing the projection display process403 e, and supplies it to the display section 104.

In step F212, the display image generation section 20 monitors an end ofa projection display mode, and returns to the single view displayprocess 403 b of step F201 if it is finished.

In the case where it is not a full circumference panorama image, thedisplay image generation section 20 proceeds from step F208 to F209,generates display image data PDdsp, by performing the scroll displayprocess 403 d, and supplies it to the display section 104.

In step F212, the display image generation section 20 monitors an end ofa projection display mode, and returns to the single view displayprocess 403 b of step F201 if it is finished.

Various types of displays are executed in the mode transitions shown inFIG. 25 by the above described processes.

That is, in the imaging apparatus 50, in the case where the panoramaimage data PD to be a display target is judged to be a fullcircumference panorama image, the control section 103 (the display imagegeneration section 20) generates display image data PDdsp by performinga display projection process, and causes a projection display to beexecuted.

In the case where it is judged not to be a full circumference panoramaimage, display image data PDdsp is generated in a scroll display processof only sequentially cutting panorama image data PD from a projectionsurface at the time of combining, without performing a displayprojection process, and a scroll display is executed.

Note that, as a modified example of this process, in the case where itis judged not to be a full circumference panorama image, a single viewdisplay process may be performed, as a display process which does notperform a display projection process. For example, it is an example inwhich an enlarged display is not performed other than for a fullcircumference panorama image, even if a specification of an enlargeddisplay is made in a single view display process.

Further, in the imaging apparatus 50 as the fifth configuration exampleof this embodiment, while an example has been set in which the imageprocessing apparatuses 1 and 2 are installed in line with the abovedescribed first configuration example, an example can naturally beassumed in which the image processing apparatuses 1 and 2 are includedin line with the above described second to fourth configurationexamples.

<8. Sixth Configuration Example (Application Example to a ComputerApparatus and Program)>

An application example to a computer apparatus and a program will bedescribed as a sixth configuration example. The processes as the abovedescribed image processing apparatuses 1 and 2 can be executed byhardware, or can be executed by software.

The program of an embodiment is a program, for example, which causes acalculation processing apparatus such as a Central Processing Unit (CPU)or a Digital Signal Processor (DSP) to execute the processes shown inthe above described embodiment.

That is, a program which implements the image processing apparatus 1 isa program which causes a calculation processing apparatus to execute aprocess, in the case where panorama image data generated by using aplurality of frame image data obtained by an imaging operation whiledisplacing an imaging direction is judged to be a full circumferencepanorama image, which associates the panorama image data withinformation which shows that it is a full circumference panorama image.

Specifically, this program may be a program which causes a calculationprocessing apparatus to execute the processes shown in FIG. 12A, FIG.14A, FIG. 16A, FIG. 18A or FIG. 24.

Further, a program which implements the image processing apparatus 2 isa program which causes a calculation processing apparatus to execute aprocess, in the case where panorama image data to be a display target isjudged to be a full circumference panorama image, which generatesdisplay image data by performing a display projection process.

Specifically, this program may be a program which causes a calculationprocessing apparatus to execute the processes shown in FIG. 12B, FIG.14B, FIG. 16B, FIG. 18B or FIG. 27.

By these programs, the above described image processing apparatuses 1and 2 can be implemented by using a calculation processing apparatus.

Programs such as these can be recorded in advance to an HDD, a ROMwithin a microcomputer having a CPU or the like as a recording mediumbuilt into a device such as a computer apparatus.

Alternatively, they can be temporarily or permanently stored (recorded)in a removable recording medium such as a flexible disc, a Compact DiscRead Only Memory (CD-ROM), a Magnet Optical (MO) disc, a DigitalVersatile Disc (DVD), a Blu-ray disc (Blu-ray (registered trademark)disc), a magnetic disc, a semiconductor memory or a memory card. Such aremovable recording medium can be provided as so-called packagesoftware.

Further, other than being installed in a personal computer or the likefrom a removable recording medium, such programs can be downloaded via anetwork such as a Local Area Network (LAN) or the internet, from adownload site.

Further, if programs such as these, they will be suitable forwide-ranging provisions of the image processing apparatuses 1 and 2 ofan embodiment. For example, by downloading a program to a personalcomputer, a portable information processing apparatus, a mobile phoneunit, a game device, a video device, a Personal Digital Assistant (PDA)or the like, this portable information processing apparatus or the likecan be set to the image processing apparatuses 1 and 2. For example, ina computer apparatus such as that shown in FIG. 28, processes the sameas those of the image processing apparatuses 1 and 2 of an embodimentcan be executed.

In FIG. 28, A CPU 71 of a computer apparatus 70 executes various typesof processes in accordance with programs stored in a ROM 72 or programsloaded from a storage section 78 to a RAM 73. Further, necessary data orthe like is also arbitrarily stored in the RAM 73 upon the CPU 71executing various types of processes.

The CPU 71, the ROM 72 and the RAM 73 are mutually connected via a bus74. Further, an input/output interface 75 is also connected to this bus74.

An input section 76 constituted by a keyboard, a mouse or the like, anoutput section 77 constituted by a display constituted by a Cathode RayTube (CRT), LCD, organic EL panel or the like, and a speaker or thelike, a storage section 78 constituted by a hard disc or the like, and acommunication section 79 constituted by a modem or the like, areconnected to the input/output interface 75. The communication section 79performs a communication process via a network which includes theinternet.

Further, a drive 80 is connected to the input/output interface 75 asnecessary, a removable media 81 such as a magnetic disc, an opticaldisk, a magneto-optical disc or a semiconductor memory is arbitrarilyincluded, and computer programs read from these are installed in thestorage section 78 as necessary.

In the case where the processes of the above described image processingapparatuses 1 and 2 are executed by software, programs constituting thissoftware are installed from a network or a recording medium.

For example, as shown in FIG. 28, this recording medium is constitutedby the removable media 81, separate from the apparatus body, which isdistributed for delivering programs to a user. For example, theremovable disc 81 is constituted by a magnetic disc (including aflexible disc), an optical disc (including a Blu-ray disc, a CD-ROM or aDVD), a magneto-optical disc (including a Mini Disc (MD)) or asemiconductor memory to which programs are recorded.

Alternatively, the recording medium is constituted by the ROM 72, a harddisc included in the storage section 78 or the like, to which programsare recorded which are delivered to a user in a state built into theapparatus body in advance.

A computer apparatus 70 such as this implements the functions of theimage processing apparatus 1, based on programs by the CPU 71, wheninputting frame image data FM#0 to FM#(n−1) of n frames for panoramaimage generation, by a reception operation by the communication section79, a reproduction operation by the drive 80 (removable media 81) or therecording section 78 or the like, and executes processes as the abovedescribed association section 11.

In this way, one panorama image data is generated from the frame imagedata FM#0 to FM#(n−1) of the n input frames.

Further, the functions of the image processing apparatus 2 areimplemented based on the programs by the CPU 71, for a display of apanorama image, and processes are executed as the above describeddisplay image generation section 20.

<9. Modified Example>

While an embodiment has been described heretofore, various types ofmodified examples can be considered for the image processing apparatusesof the present disclosure.

The first configuration example to the fourth configuration example canbe adopted in a combined manner.

For example, adding a dedicated program with metadata as an associationprocess, by combining the first configuration example and the secondconfiguration example, performing processes of making a dedicated formatwith metadata as an association process, by combining the firstconfiguration example and the fourth configuration example or the likecan be considered.

While an example which includes the functions of both the associationsection 11 and the display image generation section 20 is set as theimaging apparatus 50 of the fifth configuration example, an example canalso be considered, for example, which includes only the functions ofone of the association section 11 and the display image generationsection 20.

While there is no target of a projection display for panorama image dataPD which is not a full circumference panorama image in an embodiment, aprojection display may be performed by a specific operation. However, inthis case, since an accurate three-dimensional model is not able to bebuilt, there is the possibility for negative effects such as theoccurrence of distortions to occur in a projection display image, and sothere is the idea of allowing this as a special mode.

Besides being built into the above described imaging apparatus 50 or thecomputer apparatus 70, the image processing apparatus of the presentdisclosure is also useful for being built into a mobile phone unit, gamemachine or video machine having an imaging function, or a mobile phoneunit, a game device, a video device or an information processingapparatus not having an imaging function but having a function whichinputs frame image data.

Additionally, the present technology may also be configured as below.

-   (1) An image processing apparatus including:    -   a display image generation section configured to generate        display image data by performing a display projection process in        a case where panorama image data to be a display target is        judged to be a full circumference panorama image.-   (2) The image processing apparatus according to (1),    -   wherein the display image generation section judges the panorama        image data to be the full circumference panorama image in a case        where the panorama image data has a 360° viewing angle.-   (3) The image processing apparatus according to (1) or (2),    -   wherein the display image generation section judges the panorama        image data to be the full circumference panorama image in a case        where the panorama image data has a 360° viewing angle, and an        alignment process of both image ends is performed.-   (4) The image processing apparatus according to (1),    -   wherein the display image generation section judges the panorama        image data to be the full circumference panorama image in a case        where the panorama image data is obtained by a 360° panorama        imaging mode operation in which the panorama image data is        generated by using a plurality of frame image data obtained by        an imaging operation while displacing an imaging direction by        approximately 360°.-   (5) The image processing apparatus according to any of (1) to (4),    -   wherein the display image generation section generates display        image data in another process which does not perform the display        projection process in a case where the panorama image data to be        a display target is judged not to be the full circumference        panorama image.-   (6) The image processing apparatus according to (5),    -   wherein the another process is a process which generates display        image data for a scroll display by sequentially cutting out the        panorama image data from a projection surface at the time of        combining.-   (7) The image processing apparatus according to (5) or (6),    -   wherein the another process is a process which generates display        image data including all the panorama image data.-   (8) The image processing apparatus according to any of (1) to (7),    -   wherein the display image generation section judges whether or        not the panorama image data is the full circumference panorama        image based on metadata associated with the panorama image data.-   (9) The image processing apparatus according to any of (1) to (7),    -   wherein the display image generation section judges that the        panorama image data is the full circumference panorama image by        having a display processing program associated with the panorama        image data, and generates display image data by performing a        display projection process based on the display processing        program.-   (10) The image processing apparatus according to any of (1) to (7),    -   wherein the display image generation section judges that the        panorama image data is the full circumference panorama image by        having the panorama image data set as data of a specific format,        and generates display image data by performing the display        projection process.-   (11) The image processing apparatus according to any of (1) to (10),    -   wherein the display image generation section acquires        information showing a type of projection surface of image data        at the time of panorama image generation for the panorama image        data of a display target, and performs the display projection        process in accordance with a type of projection surface.-   (12) The image processing apparatus according to any of (1) to (11),    -   wherein the display projection process is a process which        projects, onto a plane surface, panorama image data projected        onto a projection surface at the time of panorama image        generation.-   (13) The image processing apparatus according to any of (1) to (12),    -   wherein the display image generation section sequentially        outputs display image data to which parts of the panorama image        data have been sequentially cut out in the display projection        process.

1. (canceled)
 2. An image processing method comprising: generating adisplay list view in a normal process; and upon selection of one imagein the display list view, generating display image data by performing adisplay projection process in a case where the one image is a fullcircumference image.
 3. An image processing apparatus comprising: afirst section to generate a display list view in a normal process; and asecond section to, upon selection of one image in the display list view,generate display image data by performing a display projection processin a case where the one image is a full circumference image.
 4. Theimage processing method according to claim 2, further comprising judgingthe one image to be the full circumference image in a case where the oneimage has a 360° viewing angle.
 5. The image processing method accordingto claim 2, further comprising judging the one image to be the fullcircumference image in a case where the one image has a 360° viewingangle and an alignment process of both image ends is performed.
 6. Theimage processing method according to claim 2, further comprising judgingthe one image to be the full circumference image in a case where the oneimage is obtained by a 360° imaging mode operation in which image datais generated by using a plurality of frame image data obtained by animaging operation while displacing an imaging direction by approximately360°.
 7. The image processing method according to claim 2, furthercomprising generating the display image data in another process whichdoes not perform the display projection process in a case where the oneimage is judged not to be the full circumference image.
 8. The imageprocessing method according to claim 7, wherein the another process is aprocess which generates display image data for a scroll display.
 9. Theimage processing method according to claim 2, further comprising judgingthe one image to be the full circumference image based on metadataassociated with image data of the one image.
 10. The image processingmethod according to claim 2, wherein the display projection process is aprocess which projects onto a plane surface image data.
 11. The imageprocessing apparatus according to claim 3, in which the one image is thefull circumference image in a case where the one image has a 360°viewing angle.
 12. The image processing apparatus according to claim 3,in which the one image is the full circumference image in a case wherethe one image has a 360° viewing angle and an alignment process of bothimage ends is performed.
 13. The image processing apparatus according toclaim 3, in which the one image is judged to be the full circumferenceimage in a case where the one image is obtained by a 360° imaging modeoperation in which image data is generated by using a plurality of frameimage data obtained by an imaging operation while displacing an imagingdirection by approximately 360°.
 14. The image processing apparatusaccording to claim 3, in which the display image data is generated inanother process which does not perform the display projection process ina case where the one image is judged not to be the full circumferenceimage.
 15. The image processing apparatus according to claim 14, whereinthe another process is a process which generates display image data fora scroll display.
 16. The image processing apparatus according to claim3, in which the one image is judged to be the full circumference imagebased on metadata associated with image data of the one image.
 17. Theimage processing apparatus according to claim 3, wherein the displayprojection process is a process which projects onto a plane surfaceimage data.